Information processing apparatus and method, recording medium, and program

ABSTRACT

A pattern matching section compares reference taps supplied by one area extracting section with a reference tap component of a training pair pre-stored in a training-pair storing section, selects a training pair including a reference tap component having highest similarity to the reference taps, and supplies the selected training pair to a normal-equation generating section. Training pairs stored in the training-pair storing section include a SD signal and HD signal. A prediction-coefficient determining section obtains prediction coefficients by solving normal equations generated by the normal-equation generating section. A prediction calculating section produces a HD signal by applying the prediction coefficients to prediction taps extracted by another area extracting section.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing apparatus andmethod, a recording medium, and a program. In particular, the presentinvention relates to an information processing apparatus and method, arecording medium, and a program for enabling image signals to begenerated with higher prediction accuracy. The present invention alsorelates to an information processing apparatus and method, a recordingmedium, and a program for enabling higher-quality information to beobtained in a shorter period of time. The present invention furtherrelates to an information processing apparatus and method, a recordingmedium, and a program for enabling image signals to be generated moreeasily with higher prediction accuracy.

2. Description of the Related Art

The assignee has proposed in Japanese Registered Patent Publication No.3321915 a method of classification adaptive processing for converting astandard-definition (resolution) television signal (SD signal) to ahigh-definition (resolution) image signal (HD signal). The principle ofclassification adaptive processing for generating a HD signal from a SDsignal will now be described with reference to FIGS. 1 to 4.

FIG. 1 depicts an example structure of an information processingapparatus 1 based on the known classification adaptive processing. Inthis example structure, an input SD signal is supplied to an areaextracting section 11 and an area extracting section 15. The areaextracting section 11 extracts, as class taps, predetermined pixels at apredetermined location on a preset frame of the input SD signal, andsupplies them to an ADRC processing section 12. The ADRC processingsection 12 applies 1-bit ADRC (Adaptive Dynamic Range Coding) processingto the class taps supplied by the area extracting section 11, and thensupplies the obtained 1-bit ADRC code to a class-code determiningsection 13.

The class-code determining section 13 determines a class code based onthe input 1-bit ADRC code and supplies it to a prediction coefficientstoring section 14. The relationship between the 1-bit ADRC code and theclass code is preset. The prediction coefficient storing section 14pre-stores prediction coefficients corresponding to the class code, andoutputs the prediction coefficients corresponding to the input classcode to a prediction calculating section 16.

The area extracting section 15 extracts the pixels in a preset area asprediction taps from the input SD signal, and supplies them to theprediction calculating section 16. The prediction calculating section 16generates a HD signal by applying a linear simple expression using theprediction coefficients supplied by the prediction coefficient storingsection 14 to the prediction taps supplied by the area extractingsection 15.

HD signal generation processing by the information processing apparatus1 in FIG. 1 will now be described with reference to the flowchart inFIG. 2. First in step S1, the area extracting section 11 selects onepixel of interest to be processed from the input SD signal. In step S2,the area extracting section 11 extracts class taps corresponding to thepixel of interest. Which pixels are to be set as class taps in relationto the specified pixel of interest are predetermined.

In step S3, the ADRC processing section 12 applies 1-bit ADRC processingto the class taps extracted by the area extracting section 11. In stepS4, the class-code determining section 13 determines a class code basedon the 1-bit ADRC code generated in step S3 by the ADRC processingsection 12.

In step 5, the area extracting section 15 extracts prediction taps fromthe input SD signal. The locations of the prediction taps correspondingto the pixel of interest are also preset, and the area extractingsection 15 extracts the prediction taps corresponding to the pixel ofinterest selected in step S1 and supplies them to the predictioncalculating section 16. In step S6, the prediction calculating section16 reads prediction coefficients. More specifically, the predictioncoefficient storing section 14 reads prediction coefficients stored atthe address corresponding to the class code and outputs them to theprediction calculating section 16. The prediction calculating section 16reads out the prediction coefficients.

In step S7, the prediction calculating section 16 carries out predictioncalculation. More specifically, the prediction calculating section 16applies the prediction coefficients read from the prediction coefficientstoring section 14 to the prediction taps supplied by the areaextracting section 15, based on a predetermined linear simpleexpression, to generate a HD signal. In step S8, the predictioncalculating section 16 outputs the HD signal predictively generatedthrough the processing in step S7.

In step S9, the area extracting section 11 determines whether theprocessing of all pixels has been completed. If there still remains apixel which has not been processed, the flow returns to step S1 torepeat the same processing. If it is determined in step S9 that theprocessing of all pixels has been completed, the processing ofgenerating a HD signal from a SD signal ends.

FIG. 3 depicts an example structure of an information processingapparatus 31 for producing, through training, prediction coefficientsstored in the prediction coefficient storing section 14. In thisinformation processing apparatus 31, a two-dimensional decimation filter41 generates a SD signal as a trainee signal from an input HD signal asa trainer image and supplies it to an area extracting section 42 and anarea extracting section 45. The area extracting section 42 extractsclass taps from the SD signal and supplies them to an ADRC processingsection 43. The area extracted by the area extracting section 42(positional relationships of the class taps with the pixel of interest)is the same as in the area extracting section 11 shown in FIG. 1.

The ADRC processing section 43 applies 1-bit ADRC processing to theclass taps supplied by the area extracting section 42 and outputs theADRC code to a class-code determining section 44.

The class-code determining section 44 determines a class code based onthe input ADRC code and outputs it to a normal equation generatingsection 46. The correspondence between the ADRC code and the class codein the class-code determining section 44 is the same as in theclass-code determining section 13 shown in FIG. 1.

The area extracting section 45 extracts prediction taps from the SDsignal supplied by the two-dimensional decimation filter 41 and suppliesthem to a normal equation generating section 46. The prediction areaextracted by the area extracting section 45 (positional relationships ofthe prediction taps with the pixel of interest) is the same as in thearea extracting section 15 shown in FIG. 1.

The normal equation generating section 46 generates normal equationsincluding linear simple expressions defining the relationship betweenthe SD signal and the HD signal for each class (class code), andsupplies them to a prediction coefficient determining section 47. Theprediction coefficient determining section 47 determines predictioncoefficients by solving the normal equations supplied by the normalequation generating section 46 through, for example, the least squaresmethod and supplies them to a prediction coefficient storing section 48.The prediction coefficient storing section 48 stores the predictioncoefficients supplied by the prediction coefficient determining section47.

The training processing by the information processing apparatus 31 willnow be described with reference to the flowchart shown in FIG. 4. Instep S21, the two-dimensional decimation filter 41 generates a SD signalas a trainee image by decimating every other pixel of the input HDsignal horizontally and vertically. In step S22, the area extractingsection 42 extracts class taps from the SD signal supplied by thetwo-dimensional decimation filter 41. In step S23, the ADRC processingsection 43 applies 1-bit ADRC processing to the class taps supplied bythe area extracting section 42. In step S24, the class-code determiningsection 44 determines a class code based on the ADRC code supplied bythe ADRC processing section 43.

On the other hand, in step S25 the area extracting section 45 extractsprediction taps from the SD signal supplied by the two-dimensionaldecimation filter 41 and outputs them to the normal equation generatingsection 46. In step S26, the normal equation generating section 46generates normal equations including linear simple expressions definingthe relationships between a HD signal, functioning as a trainer image,and prediction taps (SD signal), functioning as a trainee image, foreach class code supplied by the class-code determining section 44. Instep S27, the prediction coefficient determining section 47 determinesprediction coefficients by solving the normal equations generated by thenormal equation generating section 46 through, for example, the leastsquares method. In step S28, the prediction coefficient storing section48 stores the prediction coefficients supplied by the predictioncoefficient determining section 47.

In this manner, the prediction coefficients stored in the predictioncoefficient storing section 48 are used in the prediction coefficientstoring section 14 shown in FIG. 1.

As described above, a prediction coefficient set is generated throughtraining based on a prepared HD image signal and a SD image signalgenerated from the HD image signal. This training is carried out basedon many types of HD image signals. As a result, a prediction coefficientset based on the relationships between many types of HD image signalsand SD image signals is obtained.

Applying this prediction coefficient set to a received SD image signalenables a HD image signal not actually received to be predicted andgenerated. The prediction coefficient set thus obtained is based on astatistical property that is most likely to generate a signal as similarto an actual HD signal as possible in response to an input SD signal. Asa result, when a standard SD image signal is input, a HD image signalwith high accuracy on the average for each class can be predicted andgenerated.

A sufficient number of HD signals are required during training toacquire prediction coefficients through this classification adaptiveprocessing. However, some classes may not experience a sufficient amountof training depending on training materials. A class with a small amountof training cannot generate appropriate coefficients. If a HD signal isgenerated from a SD signal with prediction coefficients produced in thismanner, it is difficult to generate a HD signal with satisfactorilyenhanced image quality.

To overcome this problem, the assignee has disclosed, in JapaneseUnexamined Patent Application Publication No. 2000-78536, a method forseemingly increasing the number of training materials by intentionallyadding random numbers (noise) during training.

With the known classification adaptive processing, in which apredetermined number of classes are prepared and prediction coefficientsare generated based on the prepared classes only, images withsatisfactorily high quality are not generated in some cases.

The method of adding random numbers does not always ensure a sufficientamount of training depending on classes because the number of classes isreduced and fixed, resulting in a failure to generate images withsatisfactorily high quality.

There is another problem that a HD image signal used to generateprediction coefficients through training differs from an actuallypredicted HD image signal. This makes it difficult to ensure accurateprediction calculation processing.

A sufficient number of classes are required to overcome this problem.Unfortunately, the number of classes is limited, and if an appropriateclass is not available during training with HD image signals, a classwhich is not appropriate has to be used. This often prevents accurateprediction processing.

Moreover, the known method requires a HD image signal to generate aprediction coefficient set. As a result, the processing of generating aprediction coefficient set through training must be carried out at adifferent time or place from the processing of generating a HD imagesignal from a SD image signal by the use of the generated predictioncoefficient set. In short, the known method is problematic in thatreal-time processing from generation of coefficients to generation of HDimage signals is difficult.

SUMMARY OF THE INVENTION

The present invention is conceived in light of these circumstances, andan object of the present invention is to generate higher quality images.Another object of the present invention is to predictively generatehigh-quality images more easily and accurately. Still another object ofthe present invention is to carry out such processing in real-time.

According to one aspect of the present invention, an informationprocessing apparatus includes: a storage unit for storing a signal pairincluding a signal of a first type and a signal of a second typecorresponding to the signal of the first type; a first extraction unitfor extracting a signal in a first range from an input signal as asignal of the first type; a retrieval unit for comparing a feature ofthe extracted input signal in the first range with a feature of thesignal of the first type in the first range in the stored signal pair toretrieve a signal pair including the signal of the first type in thefirst range having a predetermined relationship with the feature of theextracted input signal in the first range; a calculation unit forcalculating a prediction coefficient based on the signal of the secondtype and the signal of the first type in a second range in the retrievedsignal pair; a second extraction unit for extracting a signal in thesecond range from the input signal; and a generation unit for generatingan output signal as a signal of the second type from the input signal inthe second range based on the calculated prediction coefficient.

The signal of the first type and the signal of the second type may beimage signals and the signal of the second type may have higherresolution than the signal of the first type.

The retrieval unit may include: a first detection unit for detecting thefeature of the input signal in the first range; a second detection unitfor detecting the feature of the stored signal of the first type in thefirst range; and a selection unit for comparing the detected feature ofthe input signal with the detected feature of the signal of the firsttype and selecting the signal pair based on a result of the comparison.

The first detection unit and the second detection unit may detect apixel value, a normalized pixel value, or a dynamic range in the firstrange as the features. The selection unit may perform the comparisonbased on a norm value, a sum of absolute differences, or a coefficientvalue of detected values.

The first detection unit and the second detection unit may detect anadaptive dynamic range coding code in the first range and the selectionunit may perform the comparison based on a coincidence of detectedcodes.

The calculation unit may generate a normal equation based on the signalof the second type and the signal of the first type in the second rangein the detected signal pair and calculate the prediction coefficient bysolving the normal equation.

According to another aspect of the present invention, an informationprocessing method includes: a first extraction step of extracting asignal in a first range from an input signal; a retrieval step ofcomparing a feature of the extracted input signal in the first rangewith a feature of a signal of a first type in the first range, thesignal of the first type and a corresponding signal of a second typebeing included in a pre-stored signal pair, to retrieve a signal pairincluding the signal of the first type in the first range having apredetermined relationship with the feature of the extracted inputsignal in the first range; a calculation step of calculating aprediction coefficient based on the signal of the second type and thesignal of the first type in a second range in the retrieved signal pair;a second extraction step of extracting a signal in the second range fromthe input signal; and a generation step of generating an output signalas a signal of the second type from the input signal in the second rangebased on the calculated prediction coefficient.

According to still another aspect of the present invention, acomputer-readable recording medium stores a program which includes: afirst extraction step of extracting a signal in a first range from aninput signal; a retrieval step of comparing a feature of the extractedinput signal in the first range with a feature of a signal of a firsttype in the first range, the signal of the first type and acorresponding signal of a second type being included in a pre-storedsignal pair, to retrieve a signal pair including the signal of the firsttype in the first range having a predetermined relationship with thefeature of the extracted input signal in the first range; a calculationstep of calculating a prediction coefficient based on the signal of thesecond type and the signal of the first type in a second range in theretrieved signal pair; a second extraction step of extracting a signalin the second range from the input signal; and a generation step ofgenerating an output signal as a signal of the second type from theinput signal in the second range based on the calculated predictioncoefficient.

According to still another aspect of the present invention, acomputer-executable program includes: a first extraction step ofextracting a signal in a first range from an input signal; a retrievalstep of comparing a feature of the extracted input signal in the firstrange with a feature of a signal of a first type in the first range, thesignal of the first type and a corresponding signal of a second typebeing included in a pre-stored signal pair, to retrieve a signal pairincluding the signal of the first type in the first range having apredetermined relationship with the feature of the extracted inputsignal in the first range; a calculation step of calculating aprediction coefficient based on the signal of the second type and thesignal of the first type in a second range in the retrieved signal pair;a second extraction step of extracting a signal in the second range fromthe input signal; and a generation step of generating an output signalas a signal of the second type from the input signal in the second rangebased on the calculated prediction coefficient.

According to another aspect of the present invention, an informationprocessing apparatus includes: a first generation unit for generating asignal of a first type from an input signal of a second type; a firstextraction unit for extracting a signal in a first range from thegenerated signal of the first type; a second extraction unit forextracting a signal in a second range from the generated signal of thefirst type; a second generation unit for generating a signal pairincluding the signal of the second type and the signal of the first typecorresponding to the signal of the second type, the signal of the firsttype being in a range defined by the logical OR between the extractedfirst range and the second range; and a storage unit for storing thesignal pair.

The signal of the first type and the signal of the second type may beimage signals, and the signal of the second type may have higherresolution than the signal of the first type.

The first generation unit may generate the signal of the first type bydecimating the signal of the second type.

According to another aspect of the present invention, an informationprocessing method includes: a first generation step of generating asignal of a first type from an input signal of a second type; a firstextraction step of extracting a signal in a first range from thegenerated signal of the first type; a second extraction step ofextracting a signal in a second range from the generated signal of thefirst type; a second generation step of generating a signal pairincluding the signal of the second type and the signal of the first typecorresponding to the signal of the second type, the signal of the firsttype being in a range defined by the logical OR between the extractedfirst range and the second range; and a storage step of storing thegenerated signal pair.

According to another aspect of the present invention, acomputer-readable recording medium stores a program which includes: afirst generation step of generating a signal of a first type from aninput signal of a second type; a first extraction step of extracting asignal in a first range from the generated signal of the first type; asecond extraction step of extracting a signal in a second range from thegenerated signal of the first type; a second generation step ofgenerating a signal pair including the signal of the second type and thesignal of the first type corresponding to the signal of the second type,the signal of the first type being in a range defined by the logical ORbetween the extracted first range and the second range; and a storagestep of storing the generated signal pair.

According to another aspect of the present invention, acomputer-executable program includes: a first generation step ofgenerating a signal of a first type from an input signal of a secondtype; a first extraction step of extracting a signal in a first rangefrom the generated signal of the first type; a second extraction step ofextracting a signal in a second range from the generated signal of thefirst type; a second generation step of generating a signal pairincluding the signal of the second type and the signal of the first typecorresponding to the signal of the second type, the signal of the firsttype being in a range defined by the logical OR between the extractedfirst range and the second range; and a storage step of storing thegenerated signal pair.

According to the present invention, the signal pair including the signalof the first type and the signal of the second type is stored, and basedon the stored signal pair the prediction coefficient is calculated. Theoutput signal is generated based on the calculated predictioncoefficient.

According to another aspect of the present invention, an informationprocessing apparatus includes: a first calculation unit for calculatinga reference value in a first range of an input signal that has apredetermined relationship with a signal of a first type and a signal ofa second type, the signals of the first and second types constituting asignal pair; a second calculation unit for calculating an average valueof the signal of the first type constituting the signal pair; and athird calculation unit for calculating a signal of a third type that hasa predetermined relationship with the signal of the first type based onthe reference value and the average value.

The input signal may be a signal of the first type.

The first calculation unit may further calculate a dynamic range in thefirst range of the input signal. The information processing apparatusmay further include: a fourth calculation unit for calculating areference value and a dynamic range in a second range of the signal ofthe second type constituting the signal pair and a normalization unitfor normalizing the signal of the first type constituting the signalpair based on the dynamic range and the reference value of the signal ofthe second type constituting the signal pair and the dynamic range ofthe input signal. The second calculation unit may calculate, as theaverage value of the signal of the first type, the average value of thenormalized signal of the first type constituting the signal pair.

The information processing apparatus may further include: a signalgeneration unit for generating the signal of the second type from theinput signal; a first extraction unit for extracting a signal in thefirst range from the input signal; and a signal-pair generation unit forgenerating the signal pair including the signal of the second type andthe corresponding signal of the first type as the input signal, wherethe signal of the second type is generated by the signal generation unitand has a predetermined relationship with the input signal in the firstrange.

The signal-pair generation unit may include: a second extraction unitfor extracting the signal in the second range from a predeterminedsearch range of the signal of the second type generated by the signalgeneration unit; a calculation unit for calculating a correlationbetween the input signal in the first range and the signal of the secondtype in the second range; a selection unit for selecting a signal of thesecond type in the second range, the signal of the second type having acorrelation equal to or higher than a threshold with respect to theinput signal in the first range, and the input signal corresponding tothe signal of the second type; and a registration unit for registeringthe input signal which is the selected signal of the first type and thesignal of the second type in the second range as the signal pair.

The reference value may be a minimum value.

The signals of the first to third types may be image signals withresolution different from one another.

The signal of the second type may be an image signal with lowerresolution than that of the signal of the first type. The signal of thethird type may be an image signal with higher resolution than that ofthe signal of the first type.

The input signal may be of the second type and the signal of the thirdtype may be identical to the signal of the first type.

The information processing apparatus may further include: a storage unitfor storing the signal pair; a first extraction unit for extracting asignal in the first range from the input signal as a signal of thesecond type; and a retrieval unit for comparing a feature of the inputsignal in the first range with a feature of the signal of the secondtype in the first range in the stored signal pair to retrieve the signalpair including the signal of the second type in the first range having apredetermined relationship with the feature of the input signal in thefirst range;

The signal of the first type and the signal of the second type may beimage signals, and the signal of the first type may have higherresolution than the signal of the second type.

The retrieval unit may include: a first detection unit for detecting thefeature of the input signal in the first range; a second detection unitfor detecting the feature of the stored signal of the second type in thefirst range; and a selection unit for comparing the detected feature ofthe input signal with the detected feature of the stored signal of thesecond type in the first range and selecting the signal pair based on aresult of the comparison.

The first detection unit and the second detection unit may detect apixel value, a normalized pixel value, or a dynamic range in the firstrange as the features. The selection unit may perform the comparisonbased on a norm value, a sum of absolute differences, or a coefficientvalue of detected values.

The first detection unit and the second detection unit may detect anadaptive dynamic range coding code in the first range and the selectionunit may perform the comparison based on a coincidence of detectedcodes.

According to another aspect of the present invention, an informationprocessing method includes: a first calculation step of calculating areference value in a first range of an input signal that has apredetermined relationship with a signal of a first type and a signal ofa second type, the signals of the first and second types constituting asignal pair; a second calculation step of calculating an average valueof the signal of the first type constituting the signal pair; and athird calculation step of calculating a signal of a third type that hasa predetermined relationship with the signal of the first type based onthe reference value and the average value.

According to another aspect of the present invention, acomputer-readable recording medium stores a program which includes: afirst calculation step of calculating a reference value in a first rangeof an input signal that has a predetermined relationship with a signalof a first type and a signal of a second type, the signals of the firstand second types constituting a signal pair; a second calculation stepof calculating an average value of the signal of the first typeconstituting the signal pair; and a third calculation step ofcalculating a signal of a third type that has a predeterminedrelationship with the signal of the first type based on the referencevalue and the average value.

According to another aspect of the present invention, acomputer-executable program includes: a first calculation step ofcalculating a reference value in a first range of an input signal thathas a predetermined relationship with a signal of a first type and asignal of a second type, the signals of the first and second typesconstituting a signal pair; a second calculation step of calculating anaverage value of the signal of the first type constituting the signalpair; and a third calculation step of calculating a signal of a thirdtype that has a predetermined relationship with the signal of the firsttype based on the reference value and the average value.

According to the present invention, the average value of the signal ofthe first type included with the signal of the second type in the signalpair is calculated, and based on the average value and the referencevalue of the signal of the first type, the signal of the third type iscalculated.

According to another aspect of the present invention, an informationprocessing apparatus includes: a signal generation unit for generating asignal of a second type from an input signal of a first type; asignal-pair generation unit for generating a signal pair including thesignal of the first type and the signal of the second type; acoefficient generation unit for generating a prediction coefficientbased on the signal pair; and a calculation unit for calculating asignal of a third type by applying the prediction coefficient to thesignal of the first type.

The signals of the first to third types may be image signals withresolution different from one another.

The signal of the second type may be an image signal with lowerresolution than that of the signal of the first type. The signal of thethird type may be an image signal with higher resolution than that ofthe signal of the first type.

The signal-pair generation unit may include: a first extraction unit forextracting a signal in a first range from the signal of the first type;a second extraction unit for extracting a signal in a second range fromthe signal of the second type; a calculation unit for calculating acorrelation between the signal of the first type in the first range andthe signal of the second type in the second range; a selection unit forselecting a signal of the second type in the second range, the signal ofthe second type having a correlation equal to or higher than a thresholdwith respect to the signal of the first type in the first range, and thesignal of the first type corresponding to the signal of the second type;and a registration unit for registering the selected signal of the firsttype and the signal of the second type in the second range as the signalpair.

The coefficient generation unit may include a normal-equation generationunit for generating a normal equation from the signal pair and acalculation unit for calculating the prediction coefficient by solvingthe normal equation.

The calculation unit may include a third extraction unit for extractinga signal of the first type in the first range from the signal of thefirst type and a product-sum calculation unit for generating the signalof the third type by calculating the product-sum of the extracted signalof the first type in the first range and the prediction coefficient.

According to another aspect of the present invention, an informationprocessing method includes: a signal generation step of generating asignal of a second type from an input signal of a first type; asignal-pair generation step of generating a signal pair including thesignal of the first type and the signal of the second type; acoefficient generation step of generating a prediction coefficient basedon the signal pair; and a calculation step of calculating a signal of athird type by applying the prediction coefficient to the signal of thefirst type.

According to another aspect of the present invention, acomputer-readable recording medium stores a program which includes: asignal generation step of generating a signal of a second type from aninput signal of a first type; a signal-pair generation step ofgenerating a signal pair including the signal of the first type and thesignal of the second type; a coefficient generation step of generating aprediction coefficient based on the signal pair; and a calculation stepof calculating a signal of a third type by applying the predictioncoefficient to the signal of the first type.

According to another aspect of the present invention, acomputer-executable program includes: a signal generation step ofgenerating a signal of a second type from an input signal of a firsttype; a signal-pair generation step of generating a signal pairincluding the signal of the first type and the signal of the secondtype; a coefficient generation step of generating a predictioncoefficient based on the signal pair; and a calculation step ofcalculating a signal of a third type by applying the predictioncoefficient to the signal of the first type.

According to the present invention, the signal of the second type isgenerated from the signal of the first type, and the signal pairincludes the signal of the first type and the signal of the second type.The prediction coefficient is generated based on the signal pair and isthen applied to the signal of the first type to calculate the signal ofthe third type.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting an example structure of a knowninformation processing apparatus;

FIG. 2 is a flowchart illustrating HD signal generation processing ofthe information processing apparatus in FIG. 1;

FIG. 3 is a block diagram depicting an example structure of a knowninformation processing apparatus for learning prediction coefficients;

FIG. 4 is a flowchart illustrating training processing of theinformation processing apparatus in FIG. 3;

FIG. 5 is a block diagram depicting an example structure of aninformation processing apparatus to which the present invention isapplied;

FIG. 6 is a block diagram depicting an example structure of the patternmatching section in FIG. 5;

FIG. 7 is a flowchart illustrating HD signal generation processing ofthe information processing apparatus in FIG. 5;

FIG. 8 is a flowchart illustrating pattern matching processing of stepS53 in FIG. 7;

FIGS. 9A to 9C illustrate pattern matching;

FIG. 10 a block diagram depicting an example structure of an informationprocessing apparatus for generating training pairs according to thepresent invention;

FIG. 11 is a flowchart illustrating training pair generation processingof the information processing apparatus in FIG. 10;

FIG. 12 illustrates reference taps;

FIG. 13 illustrates prediction taps;

FIG. 14 illustrates a SD signal for training pairs;

FIG. 15 is a block diagram depicting an example structure of a personalcomputer;

FIG. 16 is a block diagram depicting an example structure of aninformation processing apparatus to which the present invention isapplied;

FIG. 17 is a block diagram depicting an example structure of thetrainee-image generating section in FIG. 16;

FIG. 18 is a block diagram depicting an example structure of thereference-pair generating section in FIG. 16;

FIG. 19 is a block diagram depicting an example structure of theprediction calculating section in FIG. 16;

FIG. 20 is a flowchart illustrating HD image signal generationprocessing of the information processing apparatus in FIG. 16;

FIG. 21 illustrates prediction taps;

FIG. 22 is a flowchart illustrating CIF image signal generationprocessing of step S103 in FIG. 20;

FIG. 23 illustrates sub-sampling;

FIG. 24 is a flowchart illustrating reference-pair generation processingof step S104 in FIG. 20;

FIG. 25 illustrates reference taps;

FIG. 26 illustrates the relationship between prediction taps andreference taps;

FIG. 27 illustrates the relationship between prediction taps and asearch range;

FIG. 28 illustrates the relationship between reference taps andcorresponding pixel of interest;

FIG. 29 is a flowchart illustrating prediction calculation processing ofstep S105 in FIG. 20;

FIGS. 30A to 30C illustrate normalization processing;

FIGS. 31A to 31C illustrate the relationships between a CIF imagesignal, a SD image signal, and a HD image signal;

FIGS. 32A and 32B illustrate edge enhancement processing;

FIG. 33 is a block diagram illustrating an example structure of anotherinformation processing apparatus to which the present invention isapplied;

FIG. 34 is a block diagram depicting an example structure of the patternmatching section in FIG. 33;

FIG. 35 is a flowchart illustrating HD image signal generationprocessing of the information processing apparatus in FIG. 33;

FIG. 36 is a flowchart illustrating pattern matching processing of stepS203 in FIG. 35;

FIGS. 37A to 37C depict an example of reference pairs;

FIG. 38 is a block diagram depicting an example structure of aninformation processing apparatus for generating reference pairs;

FIG. 39 is a flowchart illustrating reference tap generation processingof the information processing apparatus in FIG. 38;

FIG. 40 is a block diagram depicting an example structure of aninformation processing apparatus to which the present invention isapplied;

FIG. 41 is a block diagram depicting an example structure of thetraining-pair generating section in FIG. 40;

FIG. 42 is a block diagram illustrating an example structure of thecoefficient generating section in FIG. 40;

FIG. 43 is a block diagram illustrating an example structure of theimage calculation section in FIG. 40;

FIG. 44 is a flowchart illustrating HD image signal generationprocessing of the information processing apparatus in FIG. 40;

FIG. 45 illustrates prediction taps;

FIG. 46 illustrates reference taps;

FIG. 47 illustrates the relationship between prediction taps andreference taps;

FIG. 48 illustrates the relationship between prediction taps and asearch range;

FIG. 49 is a flowchart illustrating coefficient generation processing ofstep S307 in FIG. 44;

FIG. 50 illustrates the relationship between prediction taps and a HDimage signal; and

FIGS. 51A to C illustrate the relationships between a CIF image signal,a SD image signal, and a HD image signal.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments to which the present invention is applied will now bedescribed. FIG. 5 depicts an example structure of an informationprocessing apparatus 61 to which the present invention is applied. Thisinformation processing apparatus 61 includes an area extracting section71, a pattern matching section 72, a training-pair storing section 73, anormal equation generating section 74, a prediction coefficientdetermining section 75, an area extracting section 76, and a predictioncalculating section 77.

Based on an input SD (Standard Definition) signal, the area extractingsection 71 extracts as reference taps the pixels in a predetermined area(reference area) with respect to the pixel (pixel of interest) nearestto the target location to be predicted. Through this processing, forexample, three pixels are extracted as reference taps. The referencetaps extracted by the area extracting section 71 are supplied to thepattern matching section 72.

The training-pair storing section 73 pre-stores training pairs. How tostore these training pairs will be described later with reference toFIGS. 10 and 11. These training pairs include a HD (High Definition)signal and a SD signal corresponding thereto. The SD signal includes areference tap component and a prediction tap component. The patternmatching section 72 searches the training-pair storing section 73 for atraining pair having a predetermined relationship with the referencetaps, and supplies it to the normal equation generating section 74. Thenormal equation generating section 74 generates normal equations basedon the training pair supplied by the pattern matching section 72, andoutputs them to the prediction coefficient determining section 75. Theprediction coefficient determining section 75 determines predictioncoefficients by solving the normal equations supplied by the normalequation generating section 74 through, for example, the least squaresmethod. The prediction coefficients determined by the predictioncoefficient determining section 75 are supplied to the predictioncoefficient calculating section 77.

From the input SD signal, the area extracting section 76 extracts asprediction taps the pixels in a prediction area. The relative positionalrelationships between the pixel of interest and the prediction taps arepreset. The prediction area may be the same as or different from thereference area in the area extracting section 71. The area extractingsection 76 supplies the extracted prediction taps to the predictioncalculating section 77.

The prediction calculating section 77 applies the prediction tapssupplied by the area extracting section 76 and the predictioncoefficient supplied by the prediction coefficient determining section75 to a linear simple expression to predictively generate a HD signal.

FIG. 6 depicts an example structure of the pattern matching section 72.This pattern matching section 72 includes a feature-quantity extractingsection 91, a feature-quantity extracting section 92, a comparingsection 93, a storing section 94, and a sorting section 95.

The feature-quantity extracting section 91 extracts a feature quantityof the reference taps supplied by the area extracting section 71 andsupplies it to the comparing section 93. The feature-quantity extractingsection 92 extracts a feature quantity of the reference tap component inthe training pair supplied by the training-pair storing section 73, andsupplies it to the comparing section 93. The comparing section 93compares the feature quantity supplied by the feature-quantityextracting section 91 with the feature quantity supplied by thefeature-quantity extracting section 92 to obtain the correlation betweenthe two feature quantities.

For example, the feature quantities extracted by the feature-quantityextracting section 91 and the feature-quantity extracting section 92 maybe pixel values as-is. Alternatively, such feature quantities may benormalized pixel values. A normalized pixel value in reference taps isrepresented as an intermediate pixel value relative to the maximum andminimum pixel values in the reference taps, where the maximum andminimum pixel values are set to, for example, 1 and 0, respectively.More specifically, the value obtained by subtracting the minimum valuefrom a target pixel value is divided by the difference between themaximum value and the minimum value to obtain the normalized pixel valueof the target pixel value.

Alternatively, such feature quantities may be dynamic ranges. Thedynamic range is the difference between the maximum value and theminimum value in the reference taps.

The comparing section 93 calculates a norm value, the sum of absolutedifference values, or a correlation coefficient value as a comparisonvalue for the pixel value, normalized pixel value, or dynamic range, andcarries out comparison based on the comparison value.

A norm value is calculated by the following equation, where (x1, x2, x3)represent the reference taps and (y1, y2, y3) represents the referencetap component included in the training pair.√{square root over ((x₁−y₁)²+(x₂−y₂)²+(x₃−y₃)²)}{square root over((x₁−y₁)²+(x₂−y₂)²+(x₃−y₃)²)}{square root over((x₁−y₁)²+(x₂−y₂)²+(x₃−y₃)²)}  (1)

The sum of absolute difference values is represented by Equation (2).Σ|x_(i)−y_(i)|  (2)

A correlation coefficient r is represented by Equation (3).

$\begin{matrix}{r = \frac{\sum{\left( {x_{i} - x_{M}} \right)\left( {y_{i} - y_{M}} \right)}}{\sqrt{\sum\left( {x_{i} - x_{M}} \right)^{2}}\sqrt{\sum\left( {y_{i} - y_{M}} \right)^{2}}}} & (3)\end{matrix}$

where, xM represents the average value of xi and yM represents theaverage value of yi.

In addition, the feature quantities extracted by the feature-quantityextracting section 91 and the feature-quantity extracting section 92 maybe 1-bit ADRC codes. If this is the case, the comparison by thecomparing section 93 may be based on the coincidence of 1-bit ADRCcodes. A coincidence is defined as the number of bits with equivalentvalues, where the bit at each location on one part is compared with thebit at the corresponding location on another part.

Say that the 1-bit ADRC code of the reference taps is (001) and the1-bit ADRC code of the reference tap component in the training pair isalso (001). In this case, the values of the bits at all locations in thereference taps are equal to those of the bits at the respectivelocations in the training pair, and the coincidence is “3”. On the otherhand, if the 1-bit ADRC code of the reference taps is (001) and thereference tap component in the training pair is (011), only the MSB andLSB have the equivalent values. Thus, the coincidence is “2”.

In addition to those described above, other values functioning asdeterminers of the correlation between the reference taps and thereference tap component in the training pair can be used as a comparisonvalue.

The storing section 94 stores the result of the comparison by thecomparing section 93. The sorting section 95 sorts a plurality of valuesstored in the storing section 94 in this manner, and outputs Ncomparison results sorted in descending order of correlation.Thereafter, the sorting section 95 reads out N training pairscorresponding to the top N reference tap components from thetraining-pair storing section 73, and outputs them to the normalequation generating section 74.

HD signal generation processing in the information processing apparatus61 will now be described with reference to the flowchart in FIG. 7. Instep S51, the area extracting section 71 and the area extracting section76 select a pixel of interest from the input SD signal. In step S52, thearea extracting section 71 extracts reference taps from the input SDsignal. The relative positional relationships of the reference taps withthe pixel of interest are predetermined, and the reference taps for thepixel of interest selected in step S51 are extracted in step S52. In theexample in FIGS. 9A to 9C, one pixel of interest and its neighboringeight pixels, i.e., a total of nine pixels, are set as reference taps.

In step S53, the pattern matching section 72 performs pattern matchingbetween the training pair stored in the training-pair storing section 73and the reference taps extracted in the processing of step S52. Detailsof pattern matching are shown in FIG. 8.

More specifically, in step S71, the feature-quantity extracting section91 extracts a feature quantity of the reference taps. If it is presumedthat the pixel values are extracted as the feature quantity, thefeature-quantity extracting section 91 extracts the pixel values as thefeature quantity of the reference taps supplied by the area extractingsection 71. The extracted pixel values are supplied to the comparingsection 93.

In step S72, the feature-quantity extracting section 92 extracts afeature quantity of the reference tap component in the training pair.

More specifically, the feature-quantity extracting section 92 reads outone training pair stored in the training-pair storing section 73,extracts the pixel values of the reference tap component included in thetraining pair, and outputs them to the comparing section 93. In stepS73, the comparing section 93 calculates the comparison value of thesupplied feature quantities for comparison. More specifically, thecomparing section 93 calculates as a comparison value, for example, thesum of the absolute difference values between the pixel values in thereference taps supplied by the feature-quantity extracting section 91and the pixel values in the reference tap component in the training pairsupplied by the feature-quantity extracting section 92.

As shown in FIGS. 9A to 9C, if the reference taps include, for example,a total of nine (3×3) pixel values including one pixel of interest andits neighboring eight pixels (one each above, below, left, right, upperleft, upper right, lower left, and lower right), a SD signal componentof the training pair also has a reference tap component composed of nine(3×3) pixel values. In the example of FIG. 9A, the pixel values ofreference taps RT are (100, 148, 100, 152, 200, 152, 100, 148, 100),where the pixel of interest is the pixel with the median value 200. Ifthe feature-quantity extracting section 92 extracts a first SD signal“SD1-1”, the pixel values are (200, 200, 200, 200, 200, 200, 200, 200,200). The comparing section 93 calculates the sum of the absolutedifference values as a comparison value based on the following equation.600=|100−200|+|148−200|+|100−200|+|152−200|+|200−200|+|152−200|+|100−200|+|148−200|+|100−200|  (4)

In step S74, the storing section 94 stores the comparison result. Morespecifically, the storing section 94 stores the sum of the absolutedifference values “600” calculated in the manner described above as thecomparison value.

The feature-quantity extracting section 92 then selects, for example,SD1-2 as a second SD signal and outputs it to the comparing section 93.The comparing section 93 obtains the sum of the absolute differencevalues between the pixel values in the reference taps RT and the pixelvalues in the reference tap component of the SD signal SD1-2 asdescribed above, and supplies the value to the storing section 94, whichthen stores the value.

The processing described above is repeated to calculate the same numberof sums of the absolute difference values as the number of trainingpairs stored in the training-pair storing section 73. In step S75, thesorting section 95 sorts the comparison results. More specifically, thesorting section 95 sorts the sums of the absolute difference values,i.e., the obtained comparison values in ascending order, and selects thetop 100 values. The sorting section 95 then reads out the training pairshaving the 100 reference tap components from the training-pair storingsection 73, and supplies them to the normal equation generating section74.

In this manner, as shown in, for example, FIG. 9C, 100 training pairscomposed of SD signals and HD signals are supplied to the normalequation generating section 74. In the example of FIG. 9C, 100 trainingpairs, including the training pair composed of a HD signal HD-u and a SDsignal SD-u, the training pair composed of a HD signal HD-j and a SDsignal SD-j, and the training pair composed of a HD signal HD-k and a SDsignal SD-k, are selected and supplied to the normal equation generatingsection 74.

In the example of FIGS. 9A to 9C, the reference taps are the same as theprediction taps, and hence all of the training pairs are supplied to thenormal equation generating section 74; however, the HD signals and thecorresponding prediction tap components only may be supplied.

Referring back to FIG. 7, in step S54 the normal equation generatingsection 74 generates normal equations based on the 100 training pairssupplied by the pattern matching section 72 as described above. Then instep S55, the prediction coefficient determining section 75 calculatesprediction coefficients by solving the normal equations generated in theprocessing of step S54 by, for example, the least squares method.

If the HD signal component of a training pair is “Y”, the SD signalcomponent as a prediction tap component is “X”, and the predictioncoefficients are “W”, the observation equation of a linear combinationmodel is represented by Equation (5) below.XW=Y  (5)

X, W, and Y in Equation (5) are represented as follows.

$\begin{matrix}{X = \begin{pmatrix}x_{11} & x_{12} & \ldots & x_{1n} \\x_{21} & x_{22} & \ldots & x_{2n} \\\ldots & \ldots & \ldots & \; \\x_{m\; 1} & x_{m\; 2} & \ldots & x_{m\; n}\end{pmatrix}} & (6) \\{W = \begin{pmatrix}w_{1} \\w_{2} \\\vdots \\w_{n}\end{pmatrix}} & (7) \\{Y = \begin{pmatrix}y_{1} \\y_{2} \\\vdots \\y_{m}\end{pmatrix}} & (8)\end{matrix}$

Taking into consideration the fact that the HD signal component “Y”includes errors, Equation (5) is converted into the following equation.XW=Y+E  (9)

where, E is represented by the following equation.

$\begin{matrix}{E = \begin{pmatrix}e_{1} \\e_{2} \\\vdots \\e_{m}\end{pmatrix}} & (10)\end{matrix}$

To find the most probable value of each of the coefficients wj (j=1, 2,. . . n) from Equation (9), the coefficients w1, w2, . . . , wn thatsatisfy n conditions, as shown in Equation (11), giving the minimumsquared-sum of the elements in the residual E are obtained.

$\begin{matrix}\begin{matrix}{{{e_{1}\frac{\partial e_{1}}{\partial w_{i}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{i}}} + \ldots + {e_{m}\frac{\partial e_{m}}{\partial w_{i}}}} = 0} \\\left( {{i = 0},1,2,\;\ldots\mspace{11mu},n} \right)\end{matrix} & (11)\end{matrix}$

Equation (12) shown below is obtained from Equations (9) and (10), andEquation (13) shown below is obtained from Equation (11) with conditionsj=1, 2, . . . , n.

$\begin{matrix}{{\frac{\partial e_{i}}{\partial w_{1}} = x_{i_{1}}},{\frac{\partial e_{i}}{\partial w_{2}} = x_{i_{2}}},\;\ldots\mspace{11mu},{\frac{\partial e_{i}}{\partial w_{n}} = x_{i\; n}}} & (12) \\\left( {{i = 1},2,\;\ldots\mspace{11mu},m} \right) & \; \\{{{\sum\limits_{i = 1}^{m}\;{e_{i}x_{i_{1}}}} = 0},{{\sum\limits_{i = 1}^{m}\;{e_{i}x_{i_{2}}}} = 0},\;\ldots\mspace{11mu},{{\sum\limits_{i = 1}^{m}\;{e_{i}\; x_{i\; n}}} = 0}} & (13)\end{matrix}$

Normal equations represented by Equation (14) are obtained fromEquations (9) and (13).

$\begin{matrix}\left\{ \begin{matrix}{{{\left( {\sum\limits_{i = 1}^{m}\;{x_{i_{1}}x_{i_{1}}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}\;{x_{i_{1}}x_{i_{2}}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{m}\;{x_{i_{1}}x_{i\; n}}} \right)w_{n}}} = \left( {\sum\limits_{i = 1}^{m}\;{x_{i_{1}}y_{i}}} \right)} \\{{{\left( {\sum\limits_{i = 1}^{m}\;{x_{i_{2}}x_{i_{1}}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}\;{x_{i_{2}}x_{i_{2}}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{m}\;{x_{i_{2}}x_{i\; n}}} \right)w_{n}}} = \left( {\sum\limits_{i = 1}^{m}\;{x_{i_{2}}y_{i}}} \right)} \\{\ldots\mspace{214mu}} \\{{{\left( {\sum\limits_{i = 1}^{m}\;{x_{i\; n}x_{i_{1}}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}\;{x_{i\; n}x_{i_{2}}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{m}\;{x_{i\; n}x_{i\; n}}} \right)w_{n}}} = \left( {\sum\limits_{i = 1}^{m}\;{x_{i\; n}y_{i}}} \right)}\end{matrix} \right. & (14)\end{matrix}$

The number of these normal equations (simultaneous equations) is n, thatis, the same as the number of prediction coefficients wj functioning asunknowns. Therefore, the prediction coefficients wj as the most probablevalues can be obtained by solving the normal equations. More accurately,the normal equations can be solved if the matrices of the coefficientterms of wj in Equation (14) are nonsingular. In practice, thesimultaneous equations can be solved by a method such as theGauss-Jordan's elimination (sweeping-out method).

For the known classification adaptive processing, the same number ofprediction coefficients “w1 to wn” as the number of classes areprovided. In contrast, according to the present invention, the conceptof class does not exist, and a prediction coefficient wj is generated asrequired, and hence a sufficiently large number of (virtually infinite)coefficients can be obtained. Consequently, more accurate,high-resolution HD signals can be generated.

After the prediction coefficients have been obtained as described above,in step S56 the area extracting section 76 extracts prediction taps fromthe input SD signal. The relative positional relationships of theprediction taps with the pixel of interest selected in the processing ofstep S51 are also preset. These prediction taps can be the same as thereference taps extracted in the area extracting section 71. In manycases, more prediction taps are extracted than reference taps.

In step S57, the prediction calculating section 77 carries outpredictive calculation based on the following equation.y=w1x1+w2x2+ . . . +wnxn  (15)

xi in the above-described Equation (15) is the prediction taps suppliedby the area extracting section 76, wi is the prediction coefficientssupplied by the prediction coefficient determining section 75, and y isa generated HD signal.

In step S58, the prediction calculating section 77 outputs the HD signalthat has been predicted and generated as described above.

In step S59, the area extracting section 71 and the area extractingsection 76 determine whether the processing of all pixels has beencompleted. If there is a pixel that has not been processed, the flowreturns to step S51 to repeat the same processing. If all pixels havebeen processed, the HD-signal generation processing ends.

As described above, the information processing apparatus 61 requirestraining pairs. The generation of training pairs will now be described.

FIG. 10 depicts an example structure of an information processingapparatus 121 for generating training pairs. The information processingapparatus 121 includes a two-dimensional decimation filter 131, an areaextracting section 132, an area extracting section 133, a training-pairgenerating section 134, and a training-pair storing section 135. Thetwo-dimensional decimation filter 131 generates a SD signal as a traineeimage by decimating every other pixel of a HD signal as a trainer imagehorizontally and vertically. The area extracting section 132 extractsreference taps from the SD signal supplied by the two-dimensionaldecimation filter 131. The reference taps extracted by the areaextracting section 132 are the same as the reference taps extracted bythe area extracting section 71 in FIG. 5.

The area extracting section 133 extracts prediction taps from the SDsignal supplied by the two-dimensional decimation filter 131. Theprediction taps extracted by the area extracting section 133 are alsothe same as the prediction taps extracted by the area extracting section76 in FIG. 5.

The training-pair generating section 134 receives the HD signal as atrainer image as-is, as well as the reference taps supplied by the areaextracting section 132 and the prediction taps extracted by the areaextracting section 133. The training-pair generating section 134generates a SD signal to be included in a training pair by calculatingthe logical OR between the reference taps supplied by the areaextracting section 132 and the prediction taps supplied by the areaextracting section 133. If, for example, the number of reference taps is3 and the number of prediction taps is 9, where the taps are differentfrom one another, then 12 (=3+9) taps are generated for the SD signal tobe included in the training pair. If two of the 12 taps are the same, 10taps are used for the SD signal to be included in the training pair. Ifall reference taps are included in the prediction taps, the number oftaps to be included in the training pair is 9.

The common taps are extracted by one of the area extracting section 132and the area extracting section 133, and the other extracts theremaining taps. If all reference taps are included in the predictiontaps, the physical area extracting section 132 can be omitted. If thisis the case, the area extracting section 133 includes two portions: onefor logically extracting reference taps and the other for extractingprediction taps. In this sense, the area extracting section 132 and thearea extracting section 133 exist as logical entities, even if thephysical area extracting section 132 is omitted.

The training-pair generating section 134 generates a training paircomposed of the SD signal (generated by calculating logical OR betweenthe reference taps and prediction taps) and the input HD signal,supplies the training pair to the training-pair generating section 135,which then stores it.

Training pair generation processing in the information processingapparatus 121 will now be described with reference to the flowchart inFIG. 11.

In step S91, the two-dimensional decimation filter 131 inputs a HDsignal as a trainer image. In step S92, the two-dimensional decimationfilter 131 generates a SD signal from the input HD signal. Thisprocessing can be carried out by decimating, for example, every otherpixel of the input HD signal horizontally and vertically. In step S93,the area extracting section 132 and the area extracting section 133select a pixel of interest of the input SD signal. In step S94, the areaextracting section 132 extracts reference taps with respect to the pixelof interest selected in the processing of step S93. The relativepositional relationships between the pixel of interest and the referencetaps are the same as those in the area extracting section 71 of FIG. 5.In step S95, the area extracting section 133 extracts prediction tapsfrom the input SD signal. The relative positional relationships betweenthe pixel of interest selected in step S93 and the prediction taps arethe same as those in the area extracting section 76 of FIG. 5.

In step S96, the training-pair generating section 134 generates atraining pair. More specifically, a SD signal for the training pair isgenerated by performing logical OR between the reference taps extractedby the area extracting section 132 and the prediction taps extracted bythe area extracting section 133.

For example, as shown in FIG. 12, the reference taps include threepixels: a pixel of interest P0, a pixel P1 in the left of the pixel ofinterest P0, and a pixel P2 in the right of the pixel of interest P0,where the pixels P1 and P2 are one pixel away from the pixel of interestP0. As shown in FIG. 13, the prediction taps include nine pixels: apixel of interest P0 and its neighboring eight pixels P11 to P18 (oneeach above, below, right, left, lower left, lower right, upper left, andupper right).

In this case, the logical OR between the reference taps and theprediction taps generates, as shown in FIG. 14, a total of 11 pixelsincluding pixels P0, P1, P2, and P11 to P18 for the SD signal for thetraining pair, because the pixel of interest P0 is shared. For the SDsignal for the training pair generated in this manner, the pixels P0,P1, and P2 are the reference tap component, and the pixels P0 and P11 toP18 are the prediction tap component.

Furthermore, the training-pair generating section 134 uses one pixel ofthe HD signal corresponding to the pixel of interest P0 for the HDsignal for the training pair. The training-pair generating section 134then generates the training pair from the HD signal and the SD signal.

In step S97, the training-pair storing section 135 carries out theprocessing of storing the training pair. In short, the training pairgenerated in this manner is supplied to and stored in the training-pairstoring section 135.

In step S98, the area extracting section 132 and the area extractingsection 133 determine whether the above-described processing of allpixels has been completed. If there is a pixel that has not beenprocessed, the flow returns to step S93, where the following pixel ofinterest is selected, and the newly selected pixel of interest issubjected to the same processing.

If it is determined in step S98 that all pixels have been processed, theflow proceeds to step S99, where the area extracting section 132 and thearea extracting section 133 determine whether all training data has beenprocessed. If there is training data that has not been processed, theflow returns to step S91, where a new HD signal is input, and based onthe selected HD signal, the same training pair generation processing isrepeated. If it is determined in step S99 that all training data hasbeen processed, the training pair generation processing ends.

In this manner, the training pairs stored in the training-pair storingsection 135 are used in the training-pair storing section 73 of FIG. 5.

The above-described sequence of processing can be carried out not onlywith hardware but also with software. If software is used to carry outthe above-described sequence of processing, for example, the informationprocessing apparatus can be realized by, for example, a personalcomputer as shown in FIG. 15.

In FIG. 15, a CPU (Central Processing Unit) 221 carries out variousprocessing according to programs stored in a ROM (Read Only Memory) 222or programs loaded from a storage unit 228 to a RAM (Random AccessMemory) 223. The RAM 223 also stores data required by the CPU 221 tocarry out various processing.

The CPU 221, the ROM 222, and the RAM 223 are interconnected via a bus224. An input/output interface 225 is also connected to the bus 224.

An input unit 226 including, for example, a keyboard and a mouse; anoutput unit 227 including, for example, a display section, such as a CRT(Cathode Ray Tube) and an LCD (Liquid Crystal display), and a speaker;the storage unit 228 including, for example, a hard disk; and acommunicating unit 229 including, for example, a modem are connected tothe input/output interface 225. The communicating unit 229 carries outcommunication via a network including the Internet.

A drive 230 is connected to the input/output interface 225, as required.A magnetic disk, an optical disk, a magneto-optical disk, or asemiconductor memory is mounted to the drive 230 so that computerprograms are read from the drive 230 and stored in the storage unit 228.

If the sequence of processing is to be implemented using software, aprogram constituting the software is installed from a network orrecording medium to a computer built into dedicated hardware or to, forexample, a general-purpose personal computer that requires programs tobe installed to carry out the corresponding functions.

As shown in FIG. 15, the recording medium containing the program may bea removable medium 231, such as a package medium including a magneticdisk (including a flexible disk); an optical disk (including a compactdisc-read only memory, i.e., CD-ROM and a digital versatile disk, i.e.,DVD); a magneto-optical disk (including a mini-disc, i.e., MD); or asemiconductor memory, if such a program is supplied separately from auser's computer. The recording medium may be the ROM 222 or a hard diskin the storage unit 228 if the program on the recording medium issupplied preinstalled on the user's computer.

In the present invention, the steps of programs recorded on therecording medium may or may not be followed time-sequentially in orderof described steps. Furthermore, the steps may be followed in parallelor independently from one another.

In addition, in the present description, the system represents an entireapparatus including a plurality of devices.

Another embodiment of the present invention will now described. FIG. 16depicts an example structure of an information processing apparatus 300to which the present invention is applied. The information processingapparatus 300 includes an area extracting section 311, a trainee-imagegenerating section 312, a reference-pair generating section 313, and aprediction calculating section 314.

The area extracting section 311 extracts, as prediction taps, the pixelsin the preset prediction area from an input SD (Standard Definition)image signal, and supplies them to the reference-pair generating section313 and the prediction calculating section 314. The trainee-imagegenerating section 312 generates a CIF (Common Intermediate Format)image signal as a trainee image by decimating, for example, every otherpixel of the input SD signal horizontally and vertically, and suppliesit to the reference-pair generating section 313. The reference-pairgenerating section 313 generates a signal pair as a reference pair basedon the prediction taps supplied by the area extracting section 311 andthe CIF image signal as a trainee image signal supplied by thetrainee-image generating section 312. The prediction calculating section314 generates a HD (High Definition) image signal based on theprediction taps supplied by the area extracting section 311 and thereference pair supplied by the reference-pair generating section 313.

The trainee-image generating section 312 has a structure, for example,as shown in FIG. 17. The trainee-image generating section 312 shown inFIG. 17 includes a low-pass filter 331, a phase-shifting section 332,and a sub-sampling section 333.

The low-pass filter 331 halves the horizontal and vertical bands of theinput SD image signal. According to this embodiment in which a SD imagesignal is converted into a HD image signal with resolution four timesthat of the SD signal, the low-pass filter 331 is realized as ahalf-band filter.

The phase-shifting section 332 applies phase-shifting to theband-narrowed SD image signal data supplied by the low-pass filter 331and then supplies it to the sub-sampling section 333. The sub-samplingsection 333 generates a CIF image signal by decimating horizontally andvertically every other pixel of the band-narrowed SD image signalsupplied by the phase-shifting section 332.

According to this embodiment, the prediction calculating section 314generates a HD image signal from a SD image signal. For this SD-to-HDconversion in the prediction calculating section 314, the trainee-imagegenerating section 312 carries out the inverse (HD-to-SD) conversion. Ifthe prediction calculating section 314 carries out ×1 magnificationconversion, such as HD-to-HD conversion, the phase-shifting section 332and the sub-sampling section 333 in the trainee-image generating section312 can be omitted.

FIG. 18 depicts an example structure of the reference-pair generatingsection 313. The reference-pair generating section 313 includes a tapextracting section 351, a tap-score calculating section 352, a traineeselecting section 353, a reference-pair registering section 354, and atrainer selecting section 355.

The tap extracting section 351 extracts, from the CIF image signalsupplied by the trainee-image generating section 312, reference tapswhich are pixels existing in an area (reference area) having presetrelative positional relationships with the target pixel of interest.

The relative positional relationships of the reference taps in the tapextracting section 351 with the pixel of interest are set so as to besubstantially the same as the relative positional relationships of theprediction taps in the area extracting section 311 with the pixel ofinterest (such that score calculation can be performed).

The tap-score calculating section 352 determines the correlation betweenthe reference taps extracted by the tap extracting section 351 and theprediction taps extracted by the area extracting section 311. Morespecifically, the correlation coefficient of each tap is calculated.

If the reference taps are, for example, X (X1, X2, X3, X4, X5) and theprediction taps are, for example, Y (Y1, Y2, Y3, Y4, Y5), a correlationcoefficient C is calculated based on the following equation in the samemanner as in Equation (3).

$\begin{matrix}{C = \frac{\sum\limits_{i}{\left( {X_{i} - X_{M}} \right)\left( {Y_{i} - Y_{M}} \right)}}{\sqrt{\sum\limits_{i}\left( {X_{i} - X_{M}} \right)^{2}}\sqrt{\sum\limits_{i}\left( {Y_{i} - Y_{M}} \right)^{2}}}} & (15)\end{matrix}$

where, XM and YM represent the average values of Xi and Yi,respectively.

Alternatively, the tap-score calculating section 352 may use thewaveform distance between the reference taps and the prediction taps todetermine the correlation between the two types of taps. If this is thecase, a waveform distance D is calculated based on the followingequation.

$\begin{matrix}{D = \sqrt{\sum\limits_{i}\left( {X_{i} - Y_{i}} \right)^{2}}} & (16)\end{matrix}$

If the correlation with the prediction taps is a preset threshold orhigher (if the correlation is high), the trainee selecting section 353selects the reference taps as a trainee image signal, and supplies it tothe reference-pair registering section 354 for registration. If thewaveform distance D is used as a score and is smaller than a presetthreshold, the reference taps are selected by the trainee selectingsection 353 and then supplied to and registered in the reference-pairregistering section 354.

The trainer selecting section 355, which constitutes a selection section361 with the trainee selecting section 353, selects as a trainer imagesignal the SD image signal corresponding to the reference taps selectedby the trainee selecting section 353. The reference-pair registeringsection 354 registers the reference pair, i.e., the signal pairincluding the CIF image signal as a trainee image signal supplied by thetrainee selecting section 353 and the SD image signal as a trainer imagesignal supplied by the trainer selecting section 355.

FIG. 19 depicts an example structure of the prediction calculatingsection 314. The prediction calculating section 314 includes adynamic-range (DR) calculating section 371, a minimum-value calculatingsection 372, a dynamic-range (DR) calculating section 373, aminimum-value calculating section 374, a SD-image extracting section375, a reference-pair normalizing section 376, an average calculatingsection 377, and a synthesizing section 378.

The DR calculating section 371 calculates the dynamic range, that is,the difference between the maximum value and the minimum value, of theprediction taps supplied by the area extracting section 311, and outputsit to the reference-pair normalizing section 376. The minimum-valuecalculating section 372 calculates the minimum value of the inputprediction taps and outputs it to the synthesizing section 378. This DRcalculating section 371 and the minimum-value calculating section 372can be integrated as a prediction-tap calculating section 382.

The DR calculating section 373 calculates the dynamic range, that is,the difference between the maximum value and the minimum value, of thereference pair supplied by the reference-pair generating section 313,and outputs it to the reference-pair normalizing section 376. Theminimum-value calculating section 374 calculates the minimum value ofthe input reference pair, and outputs it to the reference-pairnormalizing section 376. This DR calculating section 373 and theminimum-value calculating section 374 can be integrated as areference-pair calculating section 383. The SD-image extracting section375 extracts the SD image signal component in the input reference pair(including the SD image signal component and the CIF image signalcomponent) and outputs it to the reference-pair normalizing section 376.

The reference-pair normalizing section 376 normalizes the SD imagesignal component supplied by the SD-image extracting section 375 basedon the minimum value supplied by the minimum-value calculating section374, the dynamic range supplied by the DR calculating section 373, andthe dynamic range supplied by the DR calculating section 371. Theaverage calculating section 377 calculates the average value of the SDimage signal normalized by the reference-pair normalizing section 376,and outputs it to the synthesizing section 378. The synthesizing section378 generates a HD image signal by adding the minimum value supplied bythe minimum-value calculating section 372 to the average value suppliedby the average calculating section 377.

As described above, in this prediction calculating section 314, the SDimage signal component constituting the reference pair extracted by theSD-image extracting section 375 is normalized by the normalizing section381 including the prediction-tap calculating section 382, thereference-pair calculating section 383, and the reference-pairnormalizing section 376. The average value of the normalized SD imagesignal component is then calculated by the average calculating section377. If normalization is omitted, the normalizing section 381 can beomitted.

HD-image-signal generation processing by the information processingapparatus 300 will now be described with reference to the flowchart inFIG. 20.

First in step S101, the area extracting section 311 selects a pixel ofinterest from the input SD image signal. In step S102, the areaextracting section 311 extracts prediction taps from the input signal.In other words, pixels existing at preset locations relative to thepixel of interest selected in step S101 are extracted as predictiontaps.

FIG. 21 illustrates an example of such prediction taps. In the exampleshown in FIG. 21, nine pixel data items s1 to s9 of the SD image signalare set as prediction taps. Also in the example shown in FIG. 21, theprediction taps include pixel data S5 (pixel of interest), pixel data S4in the left of S5, pixel data S6 in the right of S5, pixel data S2 aboveS5, pixel data S1 in the left of S2, pixel data S3 in the right of S2,pixel data S8 below S5, pixel data S7 in the left of S8, and pixel dataS9 in the right of S8. The pixel data s1 to s9 are located one pixelaway from one another as shown in FIG. 21.

Next in step S103, the trainee-image generating section 312 carries outCIF-image-signal generation processing. This CIF-image-signal generationprocessing is shown as the flowchart in FIG. 22.

First in step S121, the low-pass filter 331 removes high-frequencycomponents from the input SD image signal. This prevents aliasing fromoccurring. Next in step S122, the phase-shifting section 332 appliesphase shifting to the SD image signal without high-frequency components(SD image signal supplied by the low-pass filter 331) by shifting the SDimage signal by ½ pixel horizontally and vertically. In step S123, thesub-sampling section 333 generates a CIF image signal by sub-sampling.In short, a CIF image signal is generated by decimating every otherpixel of the phase-shifted SD image signal horizontally and vertically.

More specifically, as shown in FIG. 23, after high-frequency componentsof the SD image signal have been removed by the low-pass filter 331,pixels (indicated by squares in the figure) of the SD image signal arephase-shifted by ½ pixel rightward and downward and are finallydecimated every other pixel, so that the pixels of the CIF image signalare generated as indicated by “x” in FIG. 23. In this case, one pixel ofthe CIF image signal corresponds to four pixels of the SD image signal.

Referring back to FIG. 20, in step S104, the reference-pair generatingsection 313 carries out reference tap generation processing. Details ofthis reference tap generation processing is shown as the flowchart inFIG. 24.

In step S151, the tap extracting section 351 extracts reference tapswhose center tap corresponds to one pixel in the search range of the CIFimage signal supplied by the sub-sampling section 333. These referencetaps are preset such that they have the same relative positionalrelationships as the prediction taps extracted by the area extractingsection 311. More specifically, as shown in FIG. 25, a total of ninepixel data items are extracted as reference taps from the CIF imagesignal indicated by “x” in the figure. These pixels are pixel data c5,pixel data c4 in the left of c5, pixel data c6 in the right of c5, pixeldata c2 above c5, pixel data c1 in the left of c2, pixel data c3 in theright of c2, pixel data c8 below pixel data c5, pixel data c7 in theleft of c8, and pixel data c9 in the right of c8.

Next in step S152, the tap-score calculating section 352 calculates atap score. If, for example, a correlation value is used as the score, acorrelation coefficient C between the nine prediction taps s1 to s9extracted in step S102 of FIG. 20 and the reference taps c1 to c9extracted in step S151 is calculated based on Equation (15) as shown inFIG. 26.

To ensure that the above-described calculation is carried out, theprediction taps extracted by the area extracting section 311 in stepS102 and the reference taps extracted by the tap extracting section 351in step S151 are predetermined so as to correspond to each other. Morespecifically, as shown in FIG. 26, the relative positional relationshipsbetween the reference taps and the prediction taps are determined suchthat the pixel data items C1 to C9 as the reference taps correspond tothe pixel data items S1 to S9 as the prediction taps, respectively.

Next in step S153, the trainee selecting section 353 compares the score(correlation coefficient C in this example) calculated in the processingof step S152 with a predetermined threshold. If the correlation value Cis the threshold or higher, i.e., if the CIF image signal (referencetaps) has a high correlation with the prediction taps of the HD imagesignal, the reference taps are selected and supplied to thereference-pair registering section 354. The reference-pair registeringsection 354 registers the reference taps. More specifically, the pixeldata items C1 to C9 as the reference taps extracted in step S151 shownin FIG. 25 and FIG. 26 are registered in the reference-pair registeringsection 354.

At this time, as shown in FIG. 26, the trainer selecting section 355selects the pixel of interest S5 of the SD image signal extracted instep S103 as the pixel data for training the reference taps C1 to C9(corresponding pixel of interest). This will be described later withreference to FIG. 28. The reference-pair registering section 354registers this pixel data S5 as to constitute a reference paircorresponding to the pixel data items C1 to C9.

In step S154, the tap extracting section 351 determines if the entiresearch range is searched. If not the entire search range is searched,the flow returns to step S151, where a subsequent pixel in the searchrange is selected, and reference taps having the selected pixel as itscenter are extracted. In step S152, the tap-score calculating section352 calculates the score of the reference taps selected newly in stepS151 in the same manner as described above. In step S153, the traineeselecting section 353 determines whether the score calculated in theprocessing of step S152 is the threshold or higher. If the score is thethreshold or higher, the trainee selecting section 353 supplies thereference taps to the reference-pair registering section 354, which thenstores them. If the score is below the threshold, this registrationprocessing is not carried out. The trainer selecting section 355 selectsthe pixel data item S5 of the SD image signal as the trainer data forthe pixel data items C1 to C9 selected by the trainee selecting section353 and supplies it to the reference-pair registering section 354. Thereference-pair registering section 354 registers the reference taps C1to C9 (trainee image signal) and the pixel data S5 of the SD imagesignal (trainer image signal) as a reference pair.

As shown in FIG. 27, the above-described processing is to search asearch range 113 on a CIF image plane 112 for taps correlated with theprediction taps with respect to one pixel of interest on a SD imageplane 111. The search range 113 is, for example, an area of 200×200pixels. The search range 113 is searched for reference taps having acorrelation of the threshold or higher with the prediction tapssurrounding the pixel of interest, and then the reference taps and thecorresponding pixel of interest of the SD image signal are registered asa reference pair.

More specifically, as shown in FIG. 28, the search range 113 is searchedfor reference taps having a correlation of the threshold or higher withthe prediction taps. If reference taps having a correlation of thethreshold or higher with the prediction taps are found in the searchrange 113, the pixel of interest corresponding to the reference taps isextracted in the SD image plane 111. In the example of FIG. 28, theprediction taps are (100, 150, 100, 150, 200, 150, 100, 150, 100), andthe reference taps having a correlation value of the threshold or higherin the search range 113 is (180, 190, 180, 190, 200, 190, 180, 190,180). The locations of the reference taps are any locations in thesearch range 113. They are not always the locations corresponding to theprediction taps. The pixel on the SD image plane 111, that correspondsto the tap at the center of the 3×3 reference taps (tap with a pixelvalue “200”), is extracted as the corresponding pixel of interest. Inthe example of FIG. 28, the pixel with a pixel value of “205” is thepixel of interest corresponding to the reference taps. In this case, thereference taps (180, 190, 180, 190, 200, 190, 180, 190, 180) and thecorresponding pixel of interest (205) are combined as a reference pairfor registration.

Referring back to FIG. 24, the above-described processing is repeateduntil it is determined in step S154 that the entire search range issearched. If it is determined that the search range is searched, theflow proceeds to step S155, where the tap extracting section 351determines whether all SD pixel data has been processed. If there is SDpixel data that has not been processed, the flow returns to step S151 torepeat the subsequent processing.

If it is determined in step S155 that all SD pixel data has beenprocessed, the reference tap generation processing ends.

After the reference tap generation processing described above in stepS104 of FIG. 20 has been finished, prediction calculation processing iscarried out in step S105. Details of this prediction calculationprocessing are shown as the flowchart in FIG. 29.

In step S171, the DR calculating section 371 obtains the dynamic rangeof the prediction taps extracted through the processing in step S102 ofFIG. 20. More specifically, the difference between the maximum value andthe minimum value of the pixel values of the nine taps (pixels) includedin the prediction taps is calculated as a dynamic range. In step S172,the minimum-value calculating section 372 obtains the minimum value inthe prediction taps. The minimum value is also required to calculate thedynamic range, and therefore, the minimum-value calculating section 372can be integrated with the DR calculating section 371.

In step S173, the DR calculating section 373 selects one reference pairfrom the reference pairs registered in the reference-pair registeringsection 354 through the processing in step S153, and obtains the dynamicrange, i.e., the difference between the maximum value and the minimumvalue of the CIF image signal constituting the selected reference pair.In step S174, the minimum-value calculating section 374 obtains theminimum value of the CIF image signal constituting the reference pair.Also in this case, the minimum value has already been acquired tocalculate the dynamic range, and therefore the minimum-value calculatingsection 374 can be integrated with the DR calculating section 373.

In step S175, the SD-image extracting section 375 extracts the pixelvalues of the SD image signal constituting the reference pair. Then instep S176, the reference-pair normalizing section 376 normalizes thepixel values extracted through the processing in step S175 based onEquation (17).s′j=(sj−cjmin)×DRPT/DRj  (17)

where, s′j represents the normalized SD pixel data, sj represents the SDimage data extracted in step S175, cjmin represents the minimum value ofthe CIF image data obtained in step S174, DRPT represents the dynamicrange of the prediction taps obtained in step S171, and DRj representsthe dynamic range of the CIF image data obtained in step S173.

In step S177, the reference-pair normalizing section 376 determineswhether all reference pairs have been processed. If there is a referencepair that has not been processed, the flow returns to step S173 torepeat the subsequent processing.

Details of the above-described processing will now be described withreference to FIGS. 30A to 30C. In the example shown in FIGS. 30A to 30C,the prediction taps PT are (100, 150, 100, 150, 200, 150, 100, 150, 100)as shown in FIG. 30A. The minimum value and the maximum value are “100”and “200”, respectively. Therefore, the dynamic range of the predictiontaps PT is calculated as “100” in step S171, and the minimum value ofthe prediction taps PT is calculated as “100” in step S172.

In FIG. 30B, 100 reference pairs are shown. The first reference pairincludes SD-image-signal pixel data S1 having a value “205” and CIFpixel data C1 (180, 190, 180, 190, 200, 190, 180, 190, 180). In thisreference pair, the maximum value of the CIF image signal is “200” andthe minimum value is “180”. Therefore, a dynamic range of “20” and aminimum value of “180” are obtained in steps S173 and S174,respectively. Then, in step S175, the SD pixel data S1 having a pixelvalue “205” is extracted.

In the same manner, the i-th reference pair includes CIF pixel data Ci(120, 160, 120, 160, 200, 160, 120, 160, 120) and SD pixel data Si(220). The maximum value of the CIF pixel data Ci is “200” and theminimum value of the CIF pixel data Ci is “120”. The dynamic range istherefore “80”. Thus, the SD pixel data si (220) is extracted.

The 100-th reference pair includes CIF pixel data C100 (100, 120, 100,120, 140, 120, 100, 20, 100) and SD pixel data S100 (148). The maximumvalue of the CIF pixel data C100 is “140” and the minimum value of theCIF pixel data C100 is “100”. The dynamic range is therefore “40”. TheSD pixel data S100 (148) is then extracted.

In the first reference pair, therefore, the SD pixel data S1 (205) isnormalized based on Equation (17) to be “125”, as shown in the followingequation.125=(205−180)×100/20  (18)

In the i-th reference pair, the value “220” of the SD pixel data Si isnormalized to “125” as shown in the following equation.125=(220−120)×100/80  (19)

In the 100-th reference pair, the value “148” of the SD pixel data S100is normalized to “120”, as shown in the following equation.120=(148−100)×100/40  (20)

If all reference pairs have been processed, in step S178 the averagecalculating section 377 carries out the processing of obtaining theaverage value of the normalized pixel values. In the example of FIGS.30A to 30C, the average value of the 100 normalized SD pixel data itemss′1 to s′100 (shown in FIG. 30C) is calculated.

In step S179, the synthesizing section 378 carries out the processing ofadding the average value and the minimum value of the prediction taps.More specifically, the synthesizing section 378 generates pixel data hfor the HD image signal by adding the average value A calculated throughthe processing in step S178 and the minimum value MIN of the predictiontaps obtained through the processing in step S172, as shown in thefollowing equation.h=MIN+A  (21)

In the example of FIGS. 30A to 30C, the average value of the normalizedpixel values is “124” and the minimum value of the prediction taps PT is“100”. Thus, as shown in the following equation, “224” is obtained asthe pixel data for the HD image signal corresponding to the pixel ofinterest (200) of the prediction taps PT.224=100+124  (22)

This pixel value is the pixel value for the HD image signalcorresponding to the pixel of interest.

In step S180, the synthesizing section 378 outputs the calculationresult obtained through the processing in step S179.

As shown in FIG. 28, according to the present invention, each of thecorresponding pixels of interest in reference pairs is normalized basedon the minimum value MINSD and the dynamic range DRSD of the predictiontaps on the SD image plane 111 and the minimum value MINCIF and thedynamic range DRCIF of the reference taps in the search range 113 on theCIF image plane 112 (with ¼ the resolution of the SD image plane 111)below the SD image plane 111. In short, according to this embodiment,the value obtained by subtracting the minimum value MINCIF (referencevalue) from the pixel data of the SD image signal in each reference pairis weighted by DRSD/DRCIF, and the resultant value is added to theminimum value MINSD to generate the pixel data for the HD image signal.

In other words, as shown in FIGS. 31A to 31C, the present invention isbased on a principle that the relationship between the 3×3 pixel dataitems (reference taps) of the CIF image signal constituting a referencepair and the one pixel data item (corresponding pixel of interest) ofthe SD image signal with four time the resolution is similar to therelationship between the 3×3 pixel data items of the SD image signalconstituting the prediction taps and one pixel data item for the HDimage signal with four times the resolution.

According to the present invention, the minimum value and the dynamicrange of nine pixel data items are extracted as the features of the ninepixel data items (the dynamic range can be interpreted as the maximumvalue in relation to the minimum value). The (normalized) correspondingpixel of interest in a reference pair when the minimum value and thedynamic range of reference taps are adjusted (normalized) to the minimumvalue and the dynamic range of the prediction taps can be regarded asthe HD image signal.

A reference pair and the ratio DRSD/DRCIF can be obtained without a HDimage signal, as long as a SD image signal is available. Therefore,real-time processing is possible. The use of average values does notrequire intensive computation of inverse matrix calculation as when theleast squares method is used. This enables processing to be carried outat high speed and a HD image signal to be generated easily.

Furthermore, the robustness can be enhanced compared with when the leastsquares method is used to calculate coefficients. More specifically,when coefficients are calculated by the use of the least squares method,over-fitting of coefficients may be caused (loss of robustness) if thenumber of input/output populations used for training is small.Prediction calculation based on coefficients obtained in this manner maylead to a failure in obtaining appropriate pixel values (output pixelvalues may be lost). Furthermore, the robustness can also be enhanced byaveraging normalized values to enable many reference pairs with acertain level or higher of correlation to be obtained in a limitedsearch range.

In addition, edges can be sharpened while suppressing ringing, comparedwith when the known classification adaptive processing is employed. Morespecifically, although normal high-pass processing causes ringing, whichappears as an overstressed rising edge and trailing edge as shown inFIG. 32B, according to this embodiment edges can be sharpened as shownin FIG. 32A while such ringing is suppressed.

The same advantages can also be achieved through standard generalclassification adaptive processing. However, when classification iscarried out through, for example, 1-bit ADRC processing, various inputpatterns are categorized into the same class. Although this gives thesame effect on all inputs in a class on the average, it is lesseffective due to mixed different input patterns. For example, if pixelvalues “120” or more are assigned “1” and pixel values less than “120”are assigned “0” for a 5-bit class code “00111”, gradation data (100,110, 120, 130, 140) and edge data (100, 100, 140, 140, 140) aresubjected to training prediction in the same class.

In contrast, according to this embodiment, only reference pairs similarin shape to actually processed input waveforms are used to preventringing from occurring while edges are sharpened.

Referring back to FIG. 20, after the prediction calculation processingin step S105 is completed as described above, in step S106 the areaextracting section 311 determines whether all pixels have beenprocessed. If there is a pixel that has not been processed, the flowreturns to step S101, where another pixel of the input SD image signalis set as a new pixel of interest and prediction taps in relation to thepixel of interest are extracted. Thereafter, the prediction taps areprocessed in the same manner as described above.

If it is determined in step S106 that all pixels have been processed,the HD pixel generation processing ends.

In the above-described processing, when a high-quality image with fourtimes the resolution of an input image is to be generated, an image with¼ the resolution of the input image is generated to include the originalinput image and the generated image with ¼ resolution in a referencepair. More generally, to produce an image n-times the resolution of aninput image, an image 1/n the resolution of the input image is producedto include the input image and the generated image with 1/n resolutionin a reference pair. Although “n=4” in the above-described example, n isgenerally one or a larger number.

If real-time processing is not necessary, trained reference pairs can bepre-stored for use. FIG. 33 depicts an example structure of aninformation processing apparatus 401 used when real-time processing isnot required. The information processing apparatus 401 includes an areaextracting section 411, a pattern matching section 412, a reference-pairstoring section 413, an area extracting section 711, and a predictioncalculating section 714.

Based on an input SD signal, the area extracting section 411 extracts asreference taps the pixels in a predetermined area (reference area) withrespect to a pixel of interest. Through this processing, for example,nine pixels are extracted as reference taps. The reference tapsextracted by the area extracting section 411 are supplied to the patternmatching section 412.

The reference-pair storing section 413 pre-stores reference pairs. Howto store these reference pairs will be described later with reference toFIGS. 38 and 39. The reference pairs include a HD image signal componentand a SD image signal component. The pattern matching section 412searches the reference-pair storing section 413 for a reference pairhaving a predetermined relationship with the reference taps, andsupplies it to the prediction calculating section 714.

From the input SD signal, the area extracting section 711 extracts asprediction taps the pixels in a prediction area. The relative positionalrelationships between the pixel of interest and the prediction taps arepreset. The prediction area is the same as the reference area in thearea extracting section 411. Thus, one of the area extracting section411 and the area extracting section 711 may be omitted. The areaextracting section 711 supplies the extracted prediction taps to theprediction calculating section 714.

The prediction calculating section 714 predictively calculates a HDimage signal based on the prediction taps supplied by the areaextracting section 711 and the reference pair supplied by the patternmatching section 412.

FIG. 34 depicts an example structure of the pattern matching section412. The pattern matching section 412 includes a feature-quantityextracting section 431, a feature-quantity extracting section 432, acomparing section 433, a storing section 434, and a sorting section 435.

The feature-quantity extracting section 431 extracts a feature quantityof the reference taps supplied by the area extracting section 411 andsupplies it to the comparing section 433. The feature-quantityextracting section 432 extracts a feature quantity of the SD imagesignal component in the reference pair supplied by the reference-pairstoring section 413, and supplies it to the comparing section 433. Thecomparing section 433 compares the feature quantity supplied by thefeature-quantity extracting section 431 with the feature quantitysupplied by the feature-quantity extracting section 432 to obtain thecorrelation between the two feature quantities.

For example, the feature quantities extracted by the feature-quantityextracting section 431 and the feature-quantity extracting section 432may be pixel values as-is. Alternatively, such feature quantities may benormalized pixel values. A normalized pixel value in reference taps isrepresented as an intermediate pixel value relative to the maximum andminimum pixel values in the reference taps, where the maximum andminimum pixel values are set to, for example, 1 and 0, respectively.More specifically, the value obtained by subtracting the minimum valuefrom a target pixel value is divided by the difference between themaximum value and the minimum value to obtain the normalized pixel valueof the target pixel value.

Alternatively, such feature quantities may be dynamic ranges. Thedynamic range is the difference between the maximum value and theminimum value in the reference taps.

The comparing section 433 calculates a norm value, the sum of absolutedifference values, or a correlation coefficient value as a comparisonvalue for the pixel value, normalized pixel value, or dynamic range, andcarries out comparison based on the comparison value.

A norm value is calculated by the following equation, where (x1, x2, x3. . . , x8, x9) represent the reference taps and (y1, y2, y3 . . . , y8,y9) represent the SD image signal component included in the referencepair.√{square root over ((x₁−y₁)²+(x₂−y₂)²+(x₃−y₃)²+ . . . +(x₉−y₉)²)}{squareroot over ((x₁−y₁)²+(x₂−y₂)²+(x₃−y₃)²+ . . . +(x₉−y₉)²)}{square rootover ((x₁−y₁)²+(x₂−y₂)²+(x₃−y₃)²+ . . . +(x₉−y₉)²)}{square root over((x₁−y₁)²+(x₂−y₂)²+(x₃−y₃)²+ . . . +(x₉−y₉)²)}  (23)

The sum of absolute difference values is represented by Equation (24).Σ|x_(i)−y_(i)|  (24)

A correlation coefficient C is represented by Equation (15), asdescribed above.

In addition, the feature quantities extracted by the feature-quantityextracting section 431 and the feature-quantity extracting section 432may be 1-bit ADRC codes. If this is the case, the comparison by thecomparing section 433 may be based on the coincidence of 1-bit ADRCcodes. A coincidence is defined as the number of bits with equivalentvalues, where the bit at each location on one part is compared with thebit at the corresponding location on another part.

Say that the 1-bit ADRC code of the reference taps is (000000001) andthe 1-bit ADRC code of the SD image signal component in the referencepair is also (000000001). In this case, the values of the bits at alllocations in the reference taps are equal to those of the bits at therespective locations in the SD image signal component, and thecoincidence is “9”. On the other hand, if the 1-bit ADRC code of thereference taps is (000000001) and the SD image signal component in thereference pair is (000000001), the seven bits on the MSB side and theLSB have the equivalent values. Thus, the coincidence is “8”.

In addition to those described above, other values functioning asdeterminers of the correlation between the reference taps and the SDimage signal component in the reference pair can be used as a comparisonvalue.

The storing section 434 stores the result of the comparison by thecomparing section 433. The sorting section 435 sorts a plurality ofvalues stored in the storing section 434 in this manner, and outputs Ncomparison results sorted in descending order of correlation.Thereafter, the sorting section 435 reads out N reference pairscorresponding to the top N SD image signal components from thereference-pair storing section 413, and outputs them to the predictioncalculating section 714.

HD signal generation processing in the information processing apparatus401 will now be described with reference to the flowchart in FIG. 35. Instep S201, the area extracting section 411 and the area extractingsection 711 select a pixel of interest from the input SD signal. In stepS202, the area extracting section 411 extracts reference taps from theinput SD signal. The relative positional relationships of the referencetaps with the pixel of interest are predetermined.

In step S203, the pattern matching section 412 performs pattern matchingbetween the reference pair stored in the reference-pair storing section413 and the reference taps extracted in the processing of step S202.Details of the pattern matching are shown in FIG. 36.

More specifically, in step S221, the feature-quantity extracting section431 extracts a feature quantity of the reference taps. If it is presumedthat the pixel values are extracted as the feature quantity, thefeature-quantity extracting section 431 extracts the pixel values as thefeature quantity of the reference taps supplied by the area extractingsection 411. The extracted pixel values are supplied to the comparingsection 433.

In step S222, the feature-quantity extracting section 432 extracts afeature quantity of the SD image signal component in the reference pair.

More specifically, the feature-quantity extracting section 432 reads outone reference pair stored in the reference-pair storing section 413,extracts the pixel values of the SD image signal component included inthe reference pair, and outputs them to the comparing section 433. Instep S223, the comparing section 433 calculates the comparison value ofthe supplied feature quantities for comparison. More specifically, thecomparing section 433 calculates as a comparison value, for example, thesum of the absolute difference values between the pixel values in thereference taps supplied by the feature-quantity extracting section 431and the pixel values in the SD image signal component in the referencepair supplied by the feature-quantity extracting section 432.

As shown in FIGS. 37A to 37C, if reference taps RT include, for example,a total of nine (3×3) pixel values including one pixel of interest andits neighboring eight pixels (one each above, below, left, right, upperleft, upper right, lower left, and lower right), the SD signal componentof the reference pair also has nine (3×3) pixel values. In the exampleof FIG. 97A, the pixel values of the reference taps RT are (100, 148,100, 152, 200, 152, 100, 148, 100), where the pixel of interest is thepixel with the median value 200. If the feature-quantity extractingsection 432 extracts a first SD image signal “SD1-1”, the pixel valuesare (200, 200, 200, 200, 200, 200, 200, 200, 200). The comparing section433 calculates the sum of the absolute difference values as a comparisonvalue based on the following equation.600=|100−200|+|148−200|+|100−200|+|152−200|+|200−200|+|152−200|+|100−200|+|148−200|+|100−200|  (11)

In step S224, the storing section 434 stores the comparison result. Morespecifically, the storing section 434 stores the sum of the absolutedifference values “600” calculated in the manner described above as thecomparison value.

The feature-quantity extracting section 432 then selects, for example,SD1-2 as a second SD image signal in the reference pair and output it tothe comparing section 433. The comparing section 433 calculates the sumof the absolute differences between the pixel values of the referencetaps RT and the pixel values of the SD image signal component SD1-2 inthe same manner as described above, and supplies the obtained sum to thestoring section 434, which then stores the sum.

The processing described above is repeated to calculate the same numberof sums of the absolute difference values as the number of referencepairs stored in the reference-pair storing section 413. In step S225,the sorting section 435 sorts the comparison results. More specifically,the sorting section 435 sorts the sums of the absolute differencevalues, i.e., the obtained comparison values in ascending order, andselects the top 100 values. The sorting section 435 then reads out thereference pairs having the 100 SD image signal components from thereference-pair storing section 413, and supplies them to the predictioncalculating section 714.

In this manner, as shown in, for example, FIG. 37C, 100 reference pairscomposed of SD image signals and HD image signals are supplied to theprediction calculating section 714. In the example of FIG. 37C, 100reference pairs, including the reference pair composed of a HD imagesignal HD-u and a SD image signal SD-u, the reference pair composed of aHD image signal HD-j and a SD image signal SD-j, and the reference paircomposed of a HD image signal HD-k and a SD image signal SD-k, areselected and supplied to the prediction calculating section 714.

Referring back to FIG. 35, in step S204 the area extracting section 711extracts prediction taps from the input SD image signal. The relativepositional relationships of the prediction taps with the pixel ofinterest selected in the processing of step S201 are also preset. Theseprediction taps are the same as the reference taps extracted in the areaextracting section 411.

In step S205, the prediction calculating section 714 carries outpredictive calculation processing. This processing is the same as thatshown in FIG. 29, and thus will not be described.

In step S206, the area extracting section 411 and the area extractingsection 711 determine whether the processing of all pixels has beencompleted. If there is a pixel that has not been processed, the flowreturns to step S201 to repeat the same processing. If all pixels havebeen processed, the HD-image-signal generation processing ends.

As described above, this information processing apparatus 401 requiresreference pairs to be pre-generated and pre-stored. The generation ofreference pairs will now be described.

FIG. 38 depicts an example structure of an information processingapparatus 451 for generating reference pairs. The information processingapparatus 451 includes a two-dimensional decimation filter 461, an areaextracting section 462, a reference-pair generating section 463, and areference-pair storing section 464. The two-dimensional decimationfilter 461 generates a SD image signal as a trainee image by decimatingevery other pixel of a HD image signal as a trainer image horizontallyand vertically.

The area extracting section 462 extracts reference taps from the SDimage signal supplied by the two-dimensional decimation filter 461. Thereference taps extracted by the area extracting section 462 are the sameas the reference taps extracted by the area extracting section 411(i.e., prediction taps extracted by the area extracting section 711) inFIG. 33.

The reference-pair generating section 463 receives the HD image signalas a trainer image as-is, as well as the reference taps supplied by thearea extracting section 462. The reference-pair generating section 463generates a reference pair including the reference taps supplied by thearea extracting section 462 and the corresponding input HD image signal,and supplies it to the reference-pair storing section 464 for storage.

Reference-pair generation processing in the information processingapparatus 451 will now be described with reference to the flowchart inFIG. 39.

In step S251, the two-dimensional decimation filter 461 inputs a HDimage signal as a trainer image. In step S252, the two-dimensionaldecimation filter 461 generates a SD image signal from the input HDimage signal. This processing can be carried out by decimating, forexample, every other pixel of the input HD image signal horizontally andvertically. Details of this processing are the same as those in FIG. 22.In step S253, the area extracting section 462 selects a pixel ofinterest of the input SD image signal. In step S254, the area extractingsection 462 extracts reference taps from the input SD image signal. Therelative positional relationships between the pixel of interest selectedthrough the processing of step S253 and the reference taps are the sameas those in the area extracting section 411 of FIG. 33.

In step S255, the reference-pair generating section 463 generates areference pair. More specifically, the reference taps extracted by thearea extracting section 462 in step S254 and the HD image correspondingto the reference taps that was input in step S251 generate a referencepair.

As a result of this processing, as shown in, for example, FIG. 37B, SDpixel data SD1-1 (200, 200, 200, 200, 200, 200, 200, 200, 200) as thereference taps and corresponding HD pixel data HD-1 (200) are generatedas a reference pair.

In step S256, the reference-pair storing section 464 carries out theprocessing of storing the reference pair. In short, the reference pairgenerated in this manner is supplied to and stored in the reference-pairstoring section 464.

In step S257, the area extracting section 462 determines whether theabove-described processing of all pixels has been completed. If there isa pixel that has not been processed, the flow returns to step S253,where the following pixel of interest is selected, and the newlyselected pixel of interest is subjected to the same processing.

If it is determined in step S257 that all pixels have been processed,the flow proceeds to step S258, where the area extracting section 462determines whether all training data has been processed. If there istraining data that has not been processed, the flow returns to stepS251, where a new HD image signal is input, and based on the selected HDimage signal, the same reference-pair generation processing is repeated.If it is determined in step S258 that all training data has beenprocessed, the reference-pair generation processing ends.

In this manner, the reference pairs stored in the reference-pair storingsection 464 are used by the reference-pair storing section 413 in FIG.33.

Although higher-quality images are generated in the above-describedexample, the present invention can be applied to the generation ofvarious types of signals. Furthermore, the present invention can beapplied to the processing of not only image signals but also soundsignals and other signals.

The above-described sequence of processing can be carried out not onlywith hardware but also with software. If software is used to carry outthe above-described sequence of processing, for example, the informationprocessing apparatus can be realized by, for example, a personalcomputer as shown in FIG. 15.

In the present invention, the steps of programs recorded on therecording medium may or may not be followed time-sequentially in orderof described steps. Furthermore, the steps may be followed in parallelor independently from one another.

Still another embodiment of the present invention will now be described.FIG. 40 depicts an example structure of an information processingapparatus 501 to which the present invention is applied. The informationprocessing apparatus 501 includes a trainee-image generating section511, a training-pair generating section 512, a coefficient generatingsection 513, and an image calculation section 514.

The trainee-image generating section 511 generates a CIF (CommonIntermediate Format) image signal as a trainee image by decimating, forexample, every other pixel of an input SD (Standard Definition) imagesignal horizontally and vertically, and supplies it to the training-pairgenerating section 512. The training-pair generating section 512generates a signal pair as a training pair based on the CIF image signalas a trainee image signal supplied by the trainee-image generatingsection 511 and the corresponding SD image signal.

The coefficient generating section 513 generates prediction coefficientsbased on the training pair supplied by the training-pair generatingsection 512, and outputs them to the image calculation section 514. Theimage calculation section 514 generates a HD (High Definition) imagesignal by applying the prediction coefficients to the SD image signal.

The trainee-image generating section 511 may have the same structure asdescribed with reference to FIG. 17.

FIG. 41 depicts an example structure of the training-pair generatingsection 512. The training-pair generating section 512 includes a tapextracting section 551, a tap-score calculating section 552, a traineeselecting section 553, a training-pair registering section 554, a tapextracting section 555, and a trainer selecting section 556.

The tap extracting section 551 extracts, from the CIF image signalsupplied by the trainee-image generating section 511, reference tapswhich are pixels existing in an area (reference area) having presetrelative positional relationships with the target pixel of interest.From the input SD image signal, the tap extracting section 555 extractsas prediction taps the pixels in a prediction area having apredetermined relative positional relationship with the pixel ofinterest.

The relative positional relationships of the reference taps in the tapextracting section 551 with the pixel of interest are set so as to besubstantially the same as the relative positional relationships of theprediction taps in the tap extracting section 555 with the pixel ofinterest (such that score calculation can be performed).

The tap-score calculating section 552 determines the correlation betweenthe reference taps extracted by the tap extracting section 551 and theprediction taps extracted by the tap extracting section 555. Morespecifically, the correlation coefficient of each tap is calculated.

If the reference taps are, for example, X (X1, X2, X3, X4, X5) and theprediction taps are, for example, Y (Y1, Y2, Y3, Y4, Y5), a correlationcoefficient C is calculated based on the following equation, in the samemanner as in Equation (15).

$\begin{matrix}{C = \frac{\sum\limits_{i}{\left( {X_{i} - X_{M}} \right)\left( {Y_{i} - Y_{M}} \right)}}{\sqrt{\sum\limits_{i}\left( {X_{i} - X_{M}} \right)^{2}}\sqrt{\sum\limits_{i}\left( {Y_{i} - Y_{M}} \right)}}} & (25)\end{matrix}$

where, XM and YM represent the average values of Xi and Yi,respectively.

Alternatively, the tap-score calculating section 552 may use thewaveform distance between the reference taps and the prediction taps todetermine the correlation between the two types of taps. If this is thecase, a waveform distance D is calculated based on the followingequation, in the same manner as in Equation (16).

$\begin{matrix}{D = \sqrt{\sum\limits_{i}\left( {X_{i} - Y_{i}} \right)^{2}}} & (26)\end{matrix}$

If the correlation with the prediction taps is a preset threshold orhigher, the trainee selecting section 553 selects the reference taps asa trainee image signal, and supplies it to the training-pair registeringsection 554 for registration. If the waveform distance D is used as ascore and is smaller than a preset threshold, the reference taps areselected by the trainee selecting section 553 and then supplied to andregistered in the training-pair registering section 554.

The trainer selecting section 556 selects as a trainer image signal theSD image signal corresponding to the reference taps selected by thetrainee selecting section 553. The training-pair registering section 554registers the training pair, i.e., the signal pair including the CIFimage signal as a trainee image signal supplied by the trainee selectingsection 553 and the SD image signal as a trainer image signal suppliedby the trainer selecting section 556.

FIG. 42 depicts an example structure of the coefficient generatingsection 513. The coefficient generating section 513 includes a normalequation generating section 571 and a coefficient calculating section572.

The normal equation generating section 571 generates normal equationsbased on the training pair supplied by the training-pair registeringsection 554 in the training-pair generating section 512. The coefficientcalculating section 572 calculates prediction coefficients by solvingthe normal equations generated by the normal equation generating section571 through, for example, the least squares method.

FIG. 43 depicts an example structure of the image calculation section514. The image calculation section 514 includes a prediction-tapextracting section 591 and a prediction-image generating section 592.

From the input SD image signal, the prediction-tap extracting section591 extracts as prediction taps the pixels in a preset prediction area.The relative positional relationships of these prediction taps with thepixel of interest are the same as in the tap extracting section 551 andthe tap extracting section 555 in FIG. 41. The prediction-imagegenerating section 592 applies the prediction taps extracted by theprediction-tap extracting section 591 and the prediction coefficientssupplied by the coefficient calculating section 572 to a linear simpleexpression to generate predicted pixels as a HD image signal.

HD-image-signal generation processing by the information processingapparatus 501 will now be described with reference to the flowchart inFIG. 44.

First in step S301, the trainee-image generating section 511 carries outCIF-image-signal generation processing. Details of this CIF image signalgeneration processing are the same as in the processing in FIG. 22.

After a CIF image signal has been generated in step S301, in step S302the tap extracting section 555 extracts prediction taps with respect tothe pixel of interest of the input SD image signal. As a result of thisprocessing, as shown in, for example, FIG. 45, 13 pixel data items s0 tos12 of the SD image signal are extracted as prediction taps. In theexample in FIG. 45, the predictions taps include a pixel of interest s6;pixel data items s5 and s4 located in the left of s6; pixel data itemss7 and s8 located in the right of s6; a pixel data item s2 located aboves6; a pixel data item s1 located in the left of s2; a pixel data item s3located in the right of s2; a pixel data item s0 located above s2; apixel data item s10 located below s6; a pixel data item s9 located inthe left of s10; a pixel data item s11 located in the right of s10; anda pixel data item s12 located below s10.

In step S303, the tap extracting section 551 extracts reference tapswhose center tap corresponds to one pixel in the search range of thesame CIF image signal as that supplied by the sub-sampling section 333according to the above-described embodiment. These reference taps arepreset such that they have the same relative positional relationships asthe prediction taps extracted by the tap extracting section 555. In theexample in FIG. 46, the reference taps composed of one pixel data itemc6; pixel data items c5 and c4 located in the left of c6; pixel dataitems c7 and c8 located in the right of c6; a pixel data item c2 locatedabove c6; a pixel data item c1 located in the left of c2; a pixel dataitem c3 located in the right of c2; a pixel data item c0 located abovec2; a pixel data item c10 located below c6; a pixel data item c9 locatedin the left of c10; a pixel data item c11 located in the right of c10;and a pixel data item c12 located below c10 are extracted from the CIFimage signal indicated by “x”.

Next in step S304, the tap-score calculating section 552 calculates atap score. If, for example, a correlation value is used as the score, acorrelation coefficient C between the 13 prediction taps s0 to s12extracted in step S302 and the reference taps c0 to c12 extracted instep S303 as shown in FIG. 47 is calculated based on the above-describedEquation (16).

To ensure that the above-described calculation is carried out, theprediction taps extracted by the tap extracting section 555 in step S302and the reference taps extracted by the tap extracting section 551 instep S303 are predetermined so as to correspond to each other. Morespecifically, as shown in FIGS. 45 and 46, the relative positionalrelationships between the reference taps and the prediction taps aredetermined such that the pixel data items c0 to c12 as the referencetaps correspond to the pixel data items s0 to s12 as the predictiontaps, respectively.

Next in step S305, the trainee selecting section 553 compares the score(correlation coefficient C in this example) calculated in the processingof step S304 with a predetermined threshold. If the correlation value Cis the threshold or higher, i.e., if the CIF image signal (referencetaps) has a high correlation with the prediction taps of the SD imagesignal, the reference taps are selected and supplied to thetraining-pair registering section 554. The training-pair registeringsection 554 registers the reference taps. More specifically, the pixeldata items c0 to c12 as the reference taps extracted in step S303 shownin FIG. 46 and FIG. 47 are registered in the training-pair registeringsection 554.

At this time, as shown in FIG. 47, the trainer selecting section 556selects pixel data h0 to h3 (located in the upper left, upper right,lower left, and lower right, respectively of the pixel data c6 at thecenter of the reference taps extracted in step S303) of the SD imagesignal as the pixel data of the trainer image signal corresponding tothe reference taps c0 to c12 (corresponding pixels of interest). Thetraining-pair registering section 554 registers the pixel data h0 to h3as to constitute a training pair corresponding to the pixel data c0 toc12.

In step S306, the tap extracting section 551 determines if the entiresearch range is searched. If not the entire search range is searched,the flow returns to step S303, where a subsequent pixel in the searchrange is selected, and reference taps having the selected pixel as itscenter are extracted. In step S304, the tap-score calculating section552 calculates the score of the reference taps selected newly in stepS303 in the same manner as described above. In step S305, the traineeselecting section 553 determines whether the score calculated in theprocessing of step S304 is the threshold or higher. If the score is thethreshold or higher, the trainee selecting section 553 supplies thereference taps to the training-pair registering section 554, which thenstores them. If the score is below the threshold, this registrationprocessing is not carried out. The trainer selecting section 556 selectsthe pixel data h0 to h3 of the SD image signal as the trainer data forthe pixel data c0 to c12 selected by the trainee selecting section 553and supplies the pixel data h0 to h3 to the training-pair registeringsection 554. The training-pair registering section 554 registers thereference tap c0 to c12 (trainee image signal) and the pixel data h0 toh3 of the SD image signal (trainer image signal) as a training pair.

As shown in FIG. 48, the above-described processing is to search asearch range 613 on a CIF image plane 612 for taps correlated with theprediction taps with respect to one pixel of interest on a SD imageplane 611. The search range 613 is, for example, an area of 200×200pixels. The search range 613 is searched for reference taps having acorrelation of the threshold or higher with the prediction tapssurrounding the pixel of interest, and then the reference taps and thecorresponding pixels of interest of the SD image signal are registeredas a training pair.

The above-described processing is repeated until it is determined instep S306 that the entire search range 613 is searched. If it isdetermined that the entire search range 613 is searched, the flowproceeds to step S307, where the coefficient generating section 513carries out coefficient generation processing. Details of thiscoefficient generation processing are shown as the flowchart in FIG. 49.

In step S351, the normal equation generating section 571 generatesnormal equations as shown in Equation (27) based on the training pairsupplied by the training-pair registering section 554.

$\begin{matrix}\left. \begin{matrix}\begin{matrix}\begin{matrix}{{h\; 0} = {\sum\limits_{k = 0}^{12}\;{W_{0,k} \cdot c_{k}}}} \\{{h\; 1} = {\sum\limits_{k = 0}^{12}\;{W_{1,k} \cdot c_{k}}}}\end{matrix} \\{{h\; 2} = {\sum\limits_{k = 0}^{12}\;{W_{2,k} \cdot c_{k}}}}\end{matrix} \\{{h\; 3} = {\sum\limits_{k = 0}^{12}\;{W_{3,k} \cdot c_{k}}}}\end{matrix} \right\} & (27)\end{matrix}$

The four mathematical expressions in Equations (27) represent therelationships between the pixel data h0 of the SD image signal and thepixel data c0 to c12 of the CIF image signal, the relationships betweenthe pixel data h1 of the SD image signal and the pixel data c0 to c12 ofthe CIF image signal, the relationships between the pixel data h2 of theSD image signal and the pixel data c0 to c12 of the CIF image signal,and the relationships between the pixel data h3 of the SD image signaland the pixel data c0 to c12 of the CIF image signal in FIG. 47,respectively. Thirteen or more equations that represent therelationships between the pixel data h0 and the pixel data c0 to c12 areobtained in the search range 613. Therefore, 13 unknown predictioncoefficients W0,k can be obtained. Similarly, 13 or more equations foreach of the pixel data items h1, h2, and h3, can be generated in thesearch range 613. Therefore, the prediction coefficients W1,k, W2,k, andW3,k can be obtained in the same manner.

If the normal equation generating section 571 generates normal equationsin step S351, in step S352 the coefficient calculating section 572calculates the coefficients by solving the generated normal equationsthrough, for example, the least squares method. Thus, the predictioncoefficients W0,k to W3,k in Equations (27) are obtained.

Next in step S308 of FIG. 44, the prediction-tap extracting section 591in the image calculation section 514 extracts as prediction taps thepixel data in a predetermined prediction area from the input SD imagesignal. These prediction taps are the same as the taps extracted in thetap extracting section 555 of FIG. 41. In short, as shown in FIG. 50,pixel data items s0 to s12 are selected from the input SD image signaldata as prediction taps. The shape formed by these prediction taps(positional relationships relative to the pixel of interest) is the sameas that formed by the pixel data items c0 to c12 shown in FIG. 47.

In step S309, based on the prediction coefficients, the prediction-imagegenerating section 592 carries out the processing of generating a HDimage signal from the SD image signal. More specifically, the pixel dataitems H0 to H3 for a HD image signal are generated by applying theprediction coefficients generated through the processing in step S307and the prediction taps extracted through the processing in step S308 tothe Equation (28) shown below.

$\begin{matrix}\left. \begin{matrix}\begin{matrix}\begin{matrix}{{H\; 0} = {\sum\limits_{k = 0}^{12}\;{W_{0,k} \cdot s_{k}}}} \\{{H\; 1} = {\sum\limits_{k = 0}^{12}\;{W_{1,k} \cdot s_{k}}}}\end{matrix} \\{{H\; 2} = {\sum\limits_{k = 0}^{12}\;{W_{2,k} \cdot s_{k}}}}\end{matrix} \\{{H\; 3} = {\sum\limits_{k = 0}^{12}\;{W_{3,k} \cdot s_{k}}}}\end{matrix} \right\} & (28)\end{matrix}$

As shown in FIG. 5, the data H0 to H3 indicate pixel data of the HDimage signal located, respectively, in the upper left, upper right,lower left, and lower right of the pixel data s6 as the pixel ofinterest at the center of the prediction taps s0 to s12. Predictioncoefficients W0,k to W3,k of Equation (28) are the same as theprediction coefficients W0,k to W3,k in Equation (27).

As is apparent from the comparison between FIG. 50 and FIG. 47 (or FIG.46), the relative positional relationships between the prediction tapsS0 to S12 as pixels of the SD image signal and the pixel data items H0to H3 as the HD image signal in FIG. 50 are the same as or similar tothe relative positional relationships between the pixel data items c0 toc12 of the CIF image signal and the pixel data items h0 to h3 of the SDimage signal in FIG. 47 (or FIG. 46). Furthermore, the HD image signalhas the pixel density double that of the SD image signal horizontallyand vertically, and the SD image signal also has the pixel densitydouble that of the CIF image signal horizontally and vertically.Therefore, the relationships between the pixel data c0 to c12 and thepixel data h0 to h3 are very similar to (has a significantly highcorrelation with) the relationships between the pixel data s0 to s12 andthe pixel data H0 to H3. Thus, it is expected that the predictioncoefficients W0,k to W3,k obtained based on the relationships betweenthe pixel data c0 to c12 of the CIF image signal and the pixel data h0to h3 of the SD image signal are substantially the same as or, ifdifferent, very similar to the prediction coefficients W0,k to W3,krepresenting the relationships between the pixel data s0 to s12 of theSD image signal and the pixel data H0 to H3 of the HD image signal.Accordingly, the pixels H0 to H3 of the HD image signal can be generatedby applying the prediction coefficients W0,k to W3,k to the pixel dataS0 to S12 of the SD image signal.

More specifically, as shown in FIGS. 51A to 51C, based on the fact thata field 641 of the HD image signal has the density double (horizontallyand vertically) that of a field 611 of the subordinate SD image signal,a field 612 of the CIF image signal having the density half(horizontally and vertically) that of the field 611 of the SD imagesignal is generated to obtain prediction coefficients Wi representingthe relationships between the field 611 of the SD image signal and thefield 612 of the CIF image signal, and thereafter the predictioncoefficients Wi are used to find the relationships between the field 611of SD image signal and the field 641 of the HD image signal. For thisreason, the prediction coefficients Wi can be generated without a HDimage signal as a trainer image signal, and based on that, a HD imagesignal with higher quality than that of the SD image signal can begenerated from the SD image signal.

Referring back to FIG. 44, in step S310 the image calculation section514 determines whether all SD image pixels have been processed. If thereis a SD image pixel that has not been processed, the flow returns tostep S302, where another pixel of the input SD image signal is set as anew pixel of interest and prediction taps with respect to the pixel ofinterest are extracted. Thereafter, the prediction taps are processed inthe same manner as described above.

If it is determined in step S310 that all SD pixels have been processed,the HD pixel generation processing ends.

In the above-described processing, when a high-quality image with fourtimes the resolution of an input image is to be generated, an image with¼ the resolution of the input image is generated to produce predictioncoefficients. In general, to produce an image n-times the resolution ofan input image, an image 1/n the resolution of the input image isproduced to produce prediction coefficients based on the produced image.Although “n=4” in the above-described example, n is generally one or alarger number.

Although higher-quality images are generated in the above-describedexample, the present invention can be applied to the generation ofvarious types of signals. Furthermore, the present invention can beapplied to the processing of not only image signals but also soundsignals and other signals.

The above-described sequence of processing can be carried out not onlywith hardware but also with software. If software is used to carry outthe above-described sequence of processing, for example, the informationprocessing apparatus can be realized by, for example, a personalcomputer as shown in FIG. 15.

In the present invention, the steps of programs recorded on therecording medium may or may not be followed time-sequentially in orderof described steps. Furthermore, the steps may be followed in parallelor independently from one another.

The present application contains subject matters related to JapanesePatent Application No. 2004-013888, Japanese Patent Application No.2004-013890, and Japanese Patent Application No. 2004-013891, all ofwhich were filed in Japanese Patent Office on Jan. 22, 2004 and theentire contents of which being incorporated by reference.

1. An information processing apparatus comprising: storage means forstoring a signal pair including a signal of a first type and a signal ofa second type corresponding to the signal of the first type; firstextraction means for extracting a signal in a first range from an inputsignal as a signal of the first type; retrieval means for comparing afeature of the extracted input signal in the first range with a featureof the signal of the first type in the first range in the stored signalpair to retrieve a signal pair including the signal of the first type inthe first range having a predetermined relationship with the feature ofthe extracted input signal in the first range; calculation means forcalculating a prediction coefficient based on the signal of the secondtype and the signal of the first type in a second range in the retrievedsignal pair; second extraction means for extracting a signal in thesecond range from the input signal; and generation means for generatingan output signal as a signal of the second type from the input signal inthe second range based on the calculated prediction coefficient.
 2. Theinformation processing apparatus according to claim 1, wherein thesignal of the first type and the signal of the second type are imagesignals and the signal of the second type has higher resolution thanthat of the signal of the first type.
 3. The information processingapparatus according to claim 2, wherein the retrieval means includes:first detection means for detecting the feature of the input signal inthe first range; second detection means for detecting the feature of thestored signal of the first type in the first range; and selection meansfor comparing the detected feature of the input signal with the detectedfeature of the signal of the first type and selecting the signal pairbased on a result of the comparison.
 4. The information processingapparatus according to claim 3, wherein the first detection means andthe second detection means detect a pixel value, a normalized pixelvalue, or a dynamic range in the first range as the features and theselection means performs the comparison based on a norm value, a sum ofabsolute differences, or a coefficient value of detected values.
 5. Theinformation processing apparatus according to claim 3, wherein the firstdetection means and the second detection means detect an adaptivedynamic range coding code in the first range and the selection meansperforms the comparison based on a coincidence of detected codes.
 6. Theinformation processing apparatus according to claim 1, wherein thecalculation means generates a normal equation based on the signal of thesecond type and the signal of the first type in the second range in thedetected signal pair and calculates the prediction coefficient bysolving the normal equation.
 7. An information processing method of aninformation processing apparatus for processing one or more imagesignals, the method comprising: a first extracting step, utilizing afirst extracting unit, of extracting a signal in a first range from aninput signal; a retrieval step, utilizing a retrieving unit, ofcomparing a feature of the extracted input signal in the first rangewith a feature of a signal of a first type in the first range, thesignal of the first type and a corresponding signal of a second typebeing included in a pre-stored signal pair, to retrieve a signal pairincluding the signal of the first type in the first range having apredetermined relationship with the feature of the extracted inputsignal in the first range; a calculating step, utilizing a calculatingunit, of calculating a prediction coefficient based on the signal of thesecond type and the signal of the first type in a second range in theretrieved signal pair; a second extracting step, utilizing a secondextracting unit, of extracting a signal in the second range from theinput signal; and a generating step, utilizing a generating unit, ofgenerating an output signal as a signal of the second type from theinput signal in the second range based on the calculated predictioncoefficient.
 8. A computer-readable recording medium storing a programcomprising: a first extraction step of extracting a signal in a firstrange from an input signal; a retrieval step of comparing a feature ofthe extracted input signal in the first range with a feature of a signalof a first type in the first range, the signal of the first type and acorresponding signal of a second type being included in a pre-storedsignal pair, to retrieve a signal pair including the signal of the firsttype in the first range having a predetermined relationship with thefeature of the extracted input signal in the first range; a calculationstep of calculating a prediction coefficient based on the signal of thesecond type and the signal of the first type in a second range in theretrieved signal pair; a second extraction step of extracting a signalin the second range from the input signal; and a generation step ofgenerating an output signal as a signal of the second type from theinput signal in the second range based on the calculated predictioncoefficient.
 9. An information processing apparatus comprising: firstgeneration means for generating a signal of a first type from an inputsignal of a second type; first extraction means for extracting a signalin a first range from the generated signal of the first type; secondextraction means for extracting a signal in a second range from thegenerated signal of the first type; second generation means forgenerating a signal pair including the signal of the second type and thesignal of the first type corresponding to the signal of the second type,the signal of the first type being in a range defined by the logical ORbetween the extracted first range and the second range; and storagemeans for storing the signal pair.
 10. The information processingapparatus according to claim 9, wherein the signal of the first type andthe signal of the second type are image signals and the signal of thesecond type has higher resolution than that of the signal of the firsttype.
 11. The information processing apparatus according to claim 10,wherein the first generation means generates the signal of the firsttype by decimating the signal of the second type.
 12. An informationprocessing method of an information processing apparatus for processingimage signal, the method comprising: a first generating step, utilizinga first generating unit, of generating a signal of a first type from aninput signal of a second type; a first extracting step, utilizing afirst extracting unit, of extracting a signal in a first range from thegenerated signal of the first type; a second generating step, utilizinga second generating unit, of extracting a signal in a second range fromthe generated signal of the first type; a second generating step,utilizing a second generating unit, of generating a signal pairincluding the signal of the second type and the signal of the first typecorresponding to the signal of the second type, the signal of the firsttype being in a range defined by the logical OR between the extractedfirst range and the second range; and a storage step, utilizing astorage unit, of storing the generated signal pair.
 13. Acomputer-readable recording medium storing a program comprising: a firstgeneration step of generating a signal of a first type from an inputsignal of a second type; a first extraction step of extracting a signalin a first range from the generated signal of the first type; a secondextraction step of extracting a signal in a second range from thegenerated signal of the first type; a second generation step ofgenerating a signal pair including the signal of the second type and thesignal of the first type corresponding to the signal of the second type,the signal of the first type being in a range defined by the logical ORbetween the extracted first range and the second range; and a storagestep of storing the generated signal pair.